{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow 2.0 教程-Transformer\n",
    "\n",
    "这里我们将实现一个Transformer模型，将葡萄牙语翻译为英语。Transformer的核心思想是self-attention--通过关注序列不同位置的内容获取句子的表示。\n",
    "\n",
    "Transformer的一些优点：\n",
    "- 不受限于数据的时间/空间关系\n",
    "- 可以并行计算\n",
    "- 远距离token的相互影响不需要通过很长的时间步或很深的卷积层\n",
    "- 可以学习远程依赖\n",
    "\n",
    "Transformer的缺点：\n",
    "- 对于时间序列，输出需要根据整个历史，而不是当前状态和输入，可能造成效率较低\n",
    "- 如果想要获取时间空间信息，需要额外的位置编码\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-alpha0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "# 安装tfds pip install tfds-nightly==1.0.2.dev201904090105\n",
    "import tensorflow_datasets as tfds\n",
    "import tensorflow as tf\n",
    "import tensorflow.keras.layers as layers\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "print(tf.__version__)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.数据输入pipeline\n",
    "我们将使用到Portugese-English翻译数据集。\n",
    "\n",
    "该数据集包含大约50000个训练样例，1100个验证示例和2000个测试示例。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,\n",
    "                              as_supervised=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将数据转化为subwords格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_examples, val_examples = examples['train'], examples['validation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer_en = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "(en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
    "tokenizer_pt = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "(pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "token转化测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3222, 439, 150, 7345, 1378, 2824, 2370, 7881]\n",
      "hello world, tensorflow 2\n"
     ]
    }
   ],
   "source": [
    "sample_str = 'hello world, tensorflow 2'\n",
    "tokenized_str = tokenizer_en.encode(sample_str)\n",
    "print(tokenized_str)\n",
    "original_str = tokenizer_en.decode(tokenized_str)\n",
    "print(original_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "添加start、end的token表示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode(lang1, lang2):\n",
    "    lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\n",
    "        lang1.numpy()) + [tokenizer_pt.vocab_size+1]\n",
    "    lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\n",
    "        lang2.numpy()) + [tokenizer_en.vocab_size+1]\n",
    "    return lang1, lang2\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "过滤长度超过40的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH=40\n",
    "def filter_long_sent(x, y, max_length=MAX_LENGTH):\n",
    "    return tf.logical_and(tf.size(x) <= max_length,\n",
    "                         tf.size(y) <= max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将python运算，转换为tensorflow运算节点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tf_encode(pt, en):\n",
    "    return tf.py_function(encode, [pt, en], [tf.int64, tf.int64])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构造数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUFFER_SIZE = 20000\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "# 使用.map()运行相关图操作\n",
    "train_dataset = train_examples.map(tf_encode)\n",
    "# 过滤过长的数据\n",
    "train_dataset = train_dataset.filter(filter_long_sent)\n",
    "# 使用缓存数据加速读入\n",
    "train_dataset = train_dataset.cache()\n",
    "# 打乱并获取批数据\n",
    "train_dataset = train_dataset.padded_batch(\n",
    "BATCH_SIZE, padded_shapes=([40], [40]))  # 填充为最大长度-90\n",
    "# 设置预取数据\n",
    "train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
    "\n",
    "# 验证集数据\n",
    "val_dataset = val_examples.map(tf_encode)\n",
    "val_dataset = val_dataset.filter(filter_long_sent).padded_batch(\n",
    "BATCH_SIZE, padded_shapes=([40], [40]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: id=311363, shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8214,  116,   84, ...,    0,    0,    0],\n",
       "        [8214,    7,  261, ...,    0,    0,    0],\n",
       "        [8214,  155,   39, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8214,  639,  590, ...,    0,    0,    0],\n",
       "        [8214,  204, 3441, ...,    0,    0,    0],\n",
       "        [8214,   27,   13, ...,    0,    0,    0]])>,\n",
       " <tf.Tensor: id=311364, shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8087,   83,  145, ...,    0,    0,    0],\n",
       "        [8087, 4670, 1783, ...,    0,    0,    0],\n",
       "        [8087,  169,   56, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8087,  174,   79, ...,    0,    0,    0],\n",
       "        [8087,   11,   16, ...,    0,    0,    0],\n",
       "        [8087,    4,   12, ...,    0,    0,    0]])>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "de_batch, en_batch = next(iter(train_dataset))\n",
    "de_batch, en_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.位置嵌入\n",
    "将位置编码矢量添加得到词嵌入，相同位置的词嵌入将会更接近，但并不能直接编码相对位置\n",
    "\n",
    "基于角度的位置编码方法如下：\n",
    "\n",
    "$$\\Large{PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_{model}})} $$\n",
    "$$\\Large{PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_{model}})} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_angles(pos, i, d_model):\n",
    "    # 这里的i等价与上面公式中的2i和2i+1\n",
    "    angle_rates = 1 / np.power(10000, (2*(i // 2))/ np.float32(d_model))\n",
    "    return pos * angle_rates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def positional_encoding(position, d_model):\n",
    "    angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
    "                           np.arange(d_model)[np.newaxis,:],\n",
    "                           d_model)\n",
    "    # 第2i项使用sin\n",
    "    sines = np.sin(angle_rads[:, 0::2])\n",
    "    # 第2i+1项使用cos\n",
    "    cones = np.cos(angle_rads[:, 1::2])\n",
    "    pos_encoding = np.concatenate([sines, cones], axis=-1)\n",
    "    pos_encoding = pos_encoding[np.newaxis, ...]\n",
    "    \n",
    "    return tf.cast(pos_encoding, dtype=tf.float32)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获得位置嵌入编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 50, 512)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXl4VNX5xz/n3lmTmewrSSCsAoosooJYFfd9t6K1xarVWqu1LnVrtVVrtbbazbqW/tSquFVFxAVF6wqyiMoiENaQhOzrZNZ7z++PeyeZhAADJEjwfJ7nPHebO3MyDGfOvO/5fl8hpUShUCgU3w20b7sDCoVCodhzqEFfoVAovkOoQV+hUCi+Q6hBX6FQKL5DqEFfoVAovkOoQV+hUCi+Q/TpoC+E2CCE+FoIsVQIscg+lyWEmCuEWGNvM/uyDwqFQvFtIYSYIYSoEUIs28Z1IYT4mxCiTAjxlRBiQsK16fY4uUYIMb23+rQnZvpTpZTjpJQT7eObgfeklMOB9+xjhUKh2Bf5P+DE7Vw/CRhut8uBh8GaHAN3AIcChwB39NYE+dsI75wBPGnvPwmc+S30QaFQKPocKeWHQMN2HnIG8JS0mA9kCCEKgROAuVLKBillIzCX7X95JI2jN55kO0jgHSGEBB6VUj4G5Espq+zrW4D8nm4UQlyO9c0HwnFQjtSoT0mjZGAhrnVlrNVTGOmM4C3M5YtNzYwr9NCwsY7WksG0NLZyYIGLTasqSHdouEaNZNW6SlypfkYXptC8fA2tMZPcbC+OgUMoqwnQ3tQEpoHD6yMrK5UBfjc0VRPY0kRryCAqJQ4BKbqGx+9C97hwpqeBx0/YFLRGDNpCUUJhg1jUwIxFMGNRpGlab0Nc+SwECA2haQihIXQdoelomo4QAqFhbwWaJtCEQNcFuhBoGvbWOq8J6yk1Iaynje/HXwbrPFjX7Pe18z3u8n53e/+3+gfZwfUdnN/lR27jYS3hGOlOgRQaWqSdNa2QWr6BgvH7s3JzM+m15eQduD8r1lUxOjVKa22A0OCh1FTVMm5EEVVLl2NIKB5ZzKoWB+2N9fhzcxieptH0zXqaYyaZHgf+wQNo1lKoqGsnFg5hhINoDhduv4+8dA+ZHici0EC4oYlwc5j2mElUSiTWjMohBC5N4HJpOL1OHCluNI8bzZ2CdLiQmgNTQsyURExJ1DCJGCbRmCRimBiGiTQlpimRJkgp7WaCaSLtz5aU9mdMmkjo/LzZ2y7n2IEKv5+r9GWwvk5Kmbur92tpxZJYKNnXWg4kPvgxe5xLliKgPOF4s31uW+d3m74e9A+XUlYIIfKAuUKIbxIvSiml/YWwFfYb9xiAlpIjzwn6+L/RJ3LrP25h4Pmnc0bmQTxVuJEDb7sC38/f4uObh/PclU/y3u+eYu7LH/DZDSVcc8TNnJCdysA33ueIC37HwIOn8tltE3jjgBN4v7adK08bQ97fZnL6Q/P54rVXiYXayBs9hQunTeL2Y4bAK/ez8E9v8L9v6tkSipHl1JmQ4WG/oweROaKIvBNPRO4/lbXtDj7a2Mj/VtWwZn0jDVWttFZvJNRYTTTYhhmLIE0DAM3hQnO4cHp9ODypuFLTcaam40pJxe1x4vI6cLh03B4nbq+DFI+DjBQnPo8Tv9uBz2M1r1MnxamjCYHboeFxaDg1a9+paTh10bHVhUC3f9Pp9heEJhL2sb4M4l8i8XPQ+SWhia7jb+dju47KWpJfDlr3b5ltsK2HzV3XxAnFLqIOL95NizjtA51DfvkjbvrkEybc8g6nP3QtP3vvQ8aefy9vTKrig0c+ZcVfXuBvf3icT9++k7uzx9EcNfnTjD9y5PsZLH7xGaZccRmzj3cxe8rFzNnSxjml2Ux9+k7meA/i1hmLqFm7mqYNy0jNLWHE4Yfzs1NGct7oXPTPXmDDzFcpe7OMpfVBKkNRDAkuTZDj0hmc6qS4JI38MXnkHDgE/8gRuIYdiJlVQtiXT3vUpC5oUNkapqIlxOamIJsbg1Q1BWlqDRMKRAkHo0SCMSLhGKZhEg21Y4SDmLEIRixiTTKiEfuzZiJNA2kamPbnThpGx2cwvu2+v71z/Yno0n9v3K0niIVw7Hd6sq8VSghd9wv6NLwjpaywtzXAK1ixqWr75wv2tqYv+6BQKBQ7hRAITU+q9QIVQEnCcbF9blvnd5s+G/SFEKlCCH98HzgeWAbMAuKZ6OnAa33VB4VCodh5RMcv8h21XmAW8CN7Fc8koNkOf78NHC+EyLQTuMfb53abvgzv5AOv2D//HcCzUsq3hBALgReEEJcCG4Hv92EfFAqFYuewZ/q981TiOeAoIEcIsRlrRY4TQEr5CDAHOBkoA9qBH9vXGoQQdwEL7ae6U0q5vYRw0vTZoC+lXAeM7eF8PXDMzjxXanY2l40p5o3sSfxw0/O4P3iSQfev55lHr6P2gSMZONnB+zf8luOumMztb37B6CMOZuXf76PA42DMufvz5wUbiQaaGT++EHPRHL5uDpPvdlB0xDi+rA1SU95MLNSG5nCRnp/HmKJ03K1bqF5dTlNVG20x0+qHruFPd5OSl0ZqQTaO7AICDi9NoXYa2yPUt0WIBGMd8dZ4XLV7jNRK3mpoTlfnT0Uh0BwaukOzkrEaCE3gcmjommbH5TtbPCauC6tZid3O+H18mxgT77K/jfe6pxh69zh99+NtnU8+qZt8X+IM/M10Dij4KQ9/cA8PX/8PZp2WRm3jCZz08AKe+uX3eOkhuPjpJYw75Tjeu/unHHXZIdw5ZxUDxh6BOfcJasMGEzI8xMadQvk/nsGTnssZ44sILniaFS1hfA6NvDF5yIFjWLykiZa6RkKN1QB4MwvIyEmhJN2DI1BHtHoT7TVtNIdiBAwTw85S6QK8uobPoeFOc+NK8+JM9aKl+BEuL9KVQsSQdjNpjxqEYibBiEEkZhKJmRgxK5lrGhLTTtiaZmcarOMzZmz9Oet4jNG/Y/R7GoH1f7Q3kFJesIPrErhqG9dmADN6pSMJ9HUiV6FQKPoXQqD10kx/b0QN+gqFQtGN3grv7I2oQV+hUCgS6cWY/t6IGvQVCoUiAYFAczi/7W70Gf3CZXOEX5L11Ku8ff/ZPDj9cS791OTZm44ix+Xg2oc+4zeXHsycihYKr/sddasX8pvT9+fTd9YzpTSdQdMv4sNPN+FMTWfaxBIq3pxHdTjG6DQXqYcezScbG2iuXA+AKzWdzHwfo3N9iC1raCqrpDZsEDRMXJog3amRku0lpSAbd14OZmoWbRGThmCMmpYwoWCUSDjWKZqJRrok1+JJWy1hnW986Zfu0NA0W4lrJ3R1TeCwE7cuh2Ynde1kbjx5m5jU3UaGtXsyd1uJ2DjdhVm9TbLCrO3xyH9XsXnRu7yyspbZDz3Bu0dcSN3F97Bg5guM/uQhLjhtOEtmvcWjPxjP/IYgxb+4lYol73PyscNY/ths0p0aEyYX8e76Jho3LiO9ZBRHD85i8/tLqA7HyHc7KJg4jEZXNks2NhKo2UQk0Izu8uLNzGN4vp+iNDd6aw2BilraqgM0R00iCUlWlybw6gKPx4Er1YnLn4ozLQUtNQ3T5UU63ERsJW48iRuKWUncYCRGJGZaKlxbkWvGLHVuYuLW7GGhQHdhlmIn2bPr9Pc4aqavUCgU3eivA3oyqEFfoVAoEhGi15Zs7o2oQV+hUCgSEOzbM/1+EdPf8s0mvnf1c2i//hFRKXnxn08z/K37mX7jUaz/eBYXZdWS5dJ5cr3ElZrOUSl1LGsJMfbSw2kccQyVyxaRPWwCU0vTWf/uWgwJJRMKiA06iPdX1hCsr0R3eUnJHsB+gzIoSXMS3fgNzRtbqA0bHeZZWS4dX34qqQXZ6NmFmCmZtEVM6tsjNAQihIMxouEYRiRoO2z2IMxKiONrHTF+ga5raLq91QRCiI4YvsuhdcT2rXi+FcuPx/UhLtDqFGl1NFsiZZmodY2lJ5qt7Qp9FfNPhnsfvZAH/nojN954JIXjj+XVdY2cffc8UrIHMPOq/7D/Q/8k3NrAoCUzGeBx8HpDGkYkyC++V8r8TzYzKcvLqB9OZcanG4gGminar5hS0Uj5J+UEDckwn5P0ceMoawxRWd5MpK0RMxbBlZqOP8vL8AIfOV4HZs0m2ipqaa8P0hw1OmL6icIsZ6oLd7obZ1oKeqofLdWPdKZgOtwd4qxQzCRsC7PaIwbhBGGWETMxY6YV14/H9Lt9tjrN1Mwe369kzdYUgNDQHa6kWn9EzfQVCoUiEbFvz/TVoK9QKBQJCNQ6fYVCofhOsS8P+v0ipu/QoKl8JX+fsZTrH70Ity+Tx697Ccd1fyGteARfXHUjZxxdyp+f+5LSSVOpevhPVgx+2uU8t6yaQG05g8eU4F3zEV9taibLpVNy1GjKWiQV6xqJBJrxpOfgyy9h/KAMMmSA1tVradncQkvMinv6HBrpHgcpeT6cufk4cgowvBm0hA3q2yPUt4UJB6NEQyFikWCXwilxhKZ3mK0J3Y7tO11outZRKUtowortd8Tz9R7N1hLj+l0M2BLM1uL0ZITWfa28Jrqv5+8snhK/p6fn2ll2t3hKnGt953Lam7/nq+n3Me/ek/nREQPZ+OnrXHf9eSxsDPG7LyIMPvxkPr7+cU6eOoh7Xv6a7GETGFjxGStbwxxw1ihcx/6I5V9Uobu8HHtQEeaX77GmohWXJhgwJg/H6EksqWqhobqNSKAZAHd6Dhm5qQzNSsVvthOrWm9VV2sOE7LX3IOVA/JowjZbc+Hye3D5UxApaQivH+l0E46ZHTH99qhJOGZYZmuGZbZmGlYsX8oEszX7c9WlGVvH63cVFedHrdNXKBSK7xYqvKNQKBTfGYQQaM7+uTInGdSgr1AoFIkowzWFQqH4brEvD/r9IpGbs/9w7rv/Gk4uSuPVAy7jjt9cRGUoyjkPL2DaxSfz4rvrGf/Ab9nw6dv8/JwDWPD4fI7ISWGZKOI/75ahu7z88HuDqXn9FcqDUUb4XGR+7yg+3tRIY0Ul0jRIzR1IdoGfMXl+HHXraFxdTnVrhKAh0QWkOTRS81NJLcxGzy4Afw6tYYO69gi1LWFaA1bVLCMcxIxGtjLC6m62pjk6q2bp8YpZDq1DnKVrAneiwVqC8VpipSyw9uOirUREQnI2Lsza3UTstujtqlk74tn7/8Hdd85l+rUPE7j6fCa8+SZDjzqTmwsrOWdkNk888Q73/eQQ3lhVx7jf38iajz5k3NSxrH3oEXQhGHTR91ka9FO7ajHpxSM464BCquZ+wIb2CDkuncKJpQSzhvDpmjoCtZuQpoHmcJGSXURpvp9BGR70liraN1fSWtlGQ8ToqLDm0oRttqbhdem409240lJxpaWi+TOQLi/SmZKQxDUIxwxChiXOiputGTFpi7OkbbTW1WwtkUTxVaLZmqqatWto9sKKHbX+SL8Y9BUKhWJPIYS1ii6ZluTznSiEWCWEKBNC3NzD9QeFEEvttloI0ZRwzUi4Nqs3/j4V3lEoFIpu6HrvzIeFEDrwEHAcsBlYKISYJaVcEX+MlPKXCY+/Ghif8BRBKeW4XumMjZrpKxQKRSKC3pzpHwKUSSnXSSkjwEzgjO08/gLguV74K7ZJvxj0V1SHuGDxPzl+yWyu/fWT/DT8MZecN4olr7zMA0fnETEl74r9APjxyFQ+rGtn3EUT+PP7ZWxY8iWZpQdwyogcyl7/kqAhGT4qB0ZO4Z3lW2ir3oDmcJFRmE/pwHSGZnqIlH1FQ1k9W0IxIqbEq2uW2VpeCr6iXBy5RRip2bRFLbO1mtYw4WDMKqBiC7PM6DbEWQmxfat4isP+AFmxeaGBpnctmNIltp8oyrJj+3o8br8ds7XEbZye/vGT/UAkmq19G6HN066+gh8fOxjd7eWhmSs48k+f8uotR/HWCddw9It/pGHdl5xiLselCb7IPpT2+kruOmU0n/93JRMyPLSPPZXH52+kvb6SotH7cUAGbPpgDc1Rk2E+F7mTx7O2MczaDU2EGqsBbLM1H/sXpZGf4kDWbKK1vIZATdcCKrqw4vo+h8Cd5rZahg891YeW4sd0pmA6PXZM3zJai5uthWOWMCsSMTAMsyOWb9iGa/F4fmIBlW2ZrW0vnq9EWNvGctnstUG/CChPON5sn9v6dYUYBAwG5iWc9gghFgkh5gshztzFP6kLKryjUCgUXRA7U90tRwixKOH4MSnlY7v4wtOAl6SUid/Ig6SUFUKIIcA8IcTXUsq1u/j8gBr0FQqFoit2eCdJ6qSUE7dzvQIoSTguts/1xDTgqsQTUsoKe7tOCPEBVrx/twb9fhHeUSgUij1JL4Z3FgLDhRCDhRAurIF9q1U4QoiRQCbwWcK5TCGE297PAaYAK7rfu7P0i5l+uLWJu699ic/bTiLa3sJ/zr2HCyuX4j71bsp+8RPOOqiQXzy1mIGHHEfT43cBMOjKq/n0/o20bF7NxPMuIL9mKa+uaiDdqTHomJFsjKaytqyeUHMt3sx8cor8HDo0m1xHhNbVq2je2EKLve7a59DIcTvwDfDjLijATM3GTMmkpTFKrW22FglGiYYjttladJtma5rDaZmsJZit6XaLF0SPr9PXNYFL7yyk0t1sLb4+34rhW6+zLbO1jrg+du6g47zYeo39Xm62BvCk+202PvUas8JRAmUvMuOV50g1XuD1zS1sahtKyaGnMP+nv+XUgwq57vmlZJQewPjIap5sDHH5tNH895s6PvpsE5rDxeETihBfvcM3qxvQBQzaLxvXgUewYHMz9VtaiQSacXh8ttlaCsOyU0nXokSrNtC2uY62xhABozOm79U1PJpVQMXlc+JOc+NKS0HzZ6KlphFzea3CKfYa/XgLRgyCUauIStxszTCsJqXc2mgtSbO1ngqobO9x33WEAN3RO4kqKWVMCPFz4G1AB2ZIKZcLIe4EFkkp418A04CZUkqZcPso4FEhhIk1Qb83cdXPrtIvBn2FQqHYk/RmVTgp5RxgTrdzt3c7/m0P930KjOm1jtioQV+hUCgSEKL/qm2TQQ36CoVC0Y2dSOT2O9Sgr1AoFN3Ylwf9frF6p7AonyNyUlj4/H+44bZL+LI5zEkPL+DMS87muRdWcNgjt7Nq3pv8bNqBzP/ze0zJ9rIyZSRbln2M0HQuOmoIta/OZHVbmBE+F3nHHsP/NjRQu35zh9nahCHZjCtMw1lbRuPKjWxpCtEWMxPM1ixhlm4Ls1pjguq2CFuaQjS3RQgHY8SCbRjhIEa3qlndzda6irREF7M1XddwODTcDs2qmtXNcC3RbE1PSOZ2T+DurNlaL4Yw+9xsDeCmH87giEv/TvY9P+HI+W8zcPKpPHrfPM4YlM5dD77J76+cxEvzNzPpLzeybO4HHHjMIax78E8ADP/Jhcx4by1bli8mrXgEF0woovrNt1kbiJDrdlA0ZQjBvP34eE0tLVUbMGMR3P5My2yt0M+w7BT05graN2ygtcoyWwsaVtJfF3RUzPK5HXgyPbgz/Lgz/GipfqTTMluLV81qj5q0R5M3W4OtE67dzdZ2BZXETUD0IHTcRuuPqJm+QqFQJCAQaI5+MR/eJdSgr1AoFIkIVCJXoVAovkv05pLNvY1+8RsmN1THKcveZr/jzuEm8SmXTxvNgpkv8NiJBbTFTOZ6xyNNgyv39/FuTYBDf3ww9763mmigmawhYzlrVC6rXl5M0JCMGpMHY45m9ldVHWZrmUUDOKQ0kxFZXiJrllK3qnYrszV/oa/DbC2ke2kOGx1ma6H2KOFgFCMSxIiEdmi2pjtcHWZrmkPrYrYmEoRY3c3WXPECK7tgttZ94rIjs7Xtx/+/XbM1gOnHD0Fzunjw8SVM+csS3vztcehCcPycv1K3eiHnYpmtLS08kkBtOX866wA+fu5rJmR4CE48i3VLviFQW07JAaMZnwlr31pBc9RklN9F/pSDKGsMs3pdY4fZmjezgLScdA4sySA/xQHVG2gtr6Gtqo2GiEnQsDQ18eIpPZqt+TIw3T5Mp4eQbbZmFVCx4vntEWO7Zmvxz9WOzNbMHYi2VPx++1iGa8m1/kifd1sIoQshvhBCzLaPBwshFtgFBZ63pckKhUKxdyBU5azd5RfAyoTj+4AHpZTDgEbg0j3QB4VCoUgSgaZrSbX+SJ/2WghRDJwCPGEfC+Bo4CX7IU8CveIRrVAoFL2B2Mdn+n2dyP0L8CvAbx9nA01Syph9vL2CApcDlwP40Dn0wa9Z8LtjeCR/LBdVfEHK+Q+y/JKLmTa1lEufWMiQKSdS+9ffoAso+dl1fHTXN6TmljB04khyy+fz/Mp6slw6g08cQ1nIw9rVltlaSvYA8gemM67AT57WTtOy5TSta6IxasU9fQ6N3BQn/uJ03AUFGL5cmsMGzSGDLW1halpChAIRIsEg0VAb5jbW6CdrthaP57sc+o7N1jrWE4Ou9a7ZWsdxt+faVXrTbA2g5e/P83G6m5qaV5nx6kxE7RNccfsJ3Fs1gCFHnMGHP/w15xxdytVPLSZ72ATGNC7m8cYgV18yjueW1dC4YRm6y8vxkwbCotmsLGtEFzDwgFxc46fyWXkTdZUthFsbcHh8+PMKycpPZb9cH+kiTLR8Na2bamlu2NpszefQSHfquNNceDK8uDN8ltmaL4OYy9uxRr813Gm21haK9YnZWhwVx985lDhrFxBCnArUSCkX78r9UsrHpJQTpZQTvei93DuFQqHoGSHYWhS5jdYf6cuZ/hTgdCHEyYAHSAP+CmQIIRz2bH97BQUUCoXiW6G/DujJ0GczfSnlLVLKYillKZZX9Dwp5Q+A94Fz7YdNB17rqz4oFArFziJIbpbfX78Yvg1x1k3ATCHE3cAXwL++hT4oFApFjwgBrn3YhmGP/GVSyg+klKfa++uklIdIKYdJKc+TUoZ3dH9mipPlc15k0VFHUxmKcuy9/+O668/jqdlrmPjEXyj732zuuPgg5v39Q44t9POpUUz11x9SPO4Qrjx2OJUzn2VtIMIBaW5yjz+ZuWvrqFu/Hmka+PIHM3l4DoPSXehbVlG/fD2VLeEOs7VMp45/gC3Myh+ImZpNa9ikJhBmS1OI1kCEiG22ZkYjGNHtm61ptjDLEmdpW5mtuWyzte4zCpdD267ZWiI9ma1tDyF674Owp+Y+Z/z4HurPO5Xhb73DmFO/z0OPLKTh4j/wwJ9fZMYvD+flZTVMfOheVrw7l6mnHcqK3/8ZlyYYdtVPmfH2aoxIkMzSA7hoQjGbX5vD2kCEAR4nA48cSXPGUN5dUU1L1TqkaeBJzyEz38d+JRkMy0rB0VhO2/pNtJS30hAxaLMrrLk0gUcTpDs1vC4db6YHd6Yfd6YfzZ+BdFlmayFDEo51Vs0KxKtmRWIEI0aPZmvxBQLdTde6m62ZCZ+9ZJO3KsnbFSHAoYmkWn9E2TAoFApFAoJ9O6avBn2FQqFIRPTfeH0y7LuBK4VCodgFrJm+llRL6vmEOFEIscq2nrm5h+sXCyFqhRBL7XZZwrXpQog1dpveG39fvxj0XcNHcOwVl/Hs55Vcf89pLJ/zIjcXVpLp1PnrJh+u1HTOSavhk/ogk246gdtnLceMRTj7mKGcNSqHFS98QcSU7DepiNjoo5m1uIK26g04PD5yBuYzqTQLV/UqwssXUPdNPVtCBoa0hVlunbRiP/6B+Wh5AwngojoQpiZgma0FWyOEQ51ma90LWYjusfzE4im6hqZbW90hcCSaq3UUUtE64vbdzdbAEk0lY7YWn7fE4/fbcxHsbbO1vig2UXLQUTzz0SYmXT+bT2+azCi/m7PvmUd7fSXjFv+bEq+TmS0DCLc28MdTRzF3dhnH5PkoL5nC+kVL8BcOZciEkYzQ6il7czVtMZOxGR5yvjeFr2vaWbe2gfb6SoSmk5o7kKIBfg4sSacg1YFRWUbLhqqOAipxYZbLLp6S5rTi+VYBFR+6PwPdn4Hp8mE4PIRjklBsa7O19oiBERdlxWyBVszEiMWQhrFV3L5TqGV2eW/ioq2O412I83/X6a3VO0IIHXgIOAkYDVwghBjdw0Ofl1KOs1vcwSALuAM4FDgEuEMIkbm7f1u/GPQVCoViT6EJa9KVTEuCQ4AyewFLBJgJnJFkV04A5kopG6SUjcBc4MRd+qMSUIO+QqFQdEO3LU921IAcIcSihHZ5t6cqAsoTjrdlPXOOEOIrIcRLQoiSnbx3p1CJXIVCoUggbsOQJHVSyom7+ZKvA89JKcNCiCuwjCiP3s3n3Cb9Yqb/zcY6Zk1u45IThrD41FspOfQU3jrhGn50zRT+/M/3GH/aSSz71S0UeBz4Lv41Kz/6gszSA7j04GLEh8+woLyFEq+T4WdPZmFVO5tW1RFubSAlZwBDhmYxJi+V2JolNHy1irr1nWZraQ6d7EwP/uJMXEWDMH05NIUNtrSGqWoJUdUUJNQeJdIeIBrcvtma0LStzNbiJmu6w7JpjRdNcTl02zytM77fk9laYkH0+LZ70ZTEcHr32Lomul7fXbO13Y3c70zof9lNI/n170+h+usP+eSw47h49p2s/3gWh077Pi9c/i+m/fww7nhiIQMnnUz2xzNY3RbhoKuP4C8fbaBl82qKDxzPD48aQmTeM3xR0YrPoVFyeDHamKP437p66ivqiAaacaWmk56fw4RBmYzO9eELNxDdsJKWjQ00tIRpiVlma7oAr26t0Xenu/BkevBkpuLJ8KP5MhAp6Uh3KqGYScgwaYtYBmtt4ViH2VowYhCLGpgxE9OQmFJuZbZmJpitqfh839GLitwKoCTheCvrGSllfYJe6QngoGTv3RX6xaCvUCgUe4peFmctBIbbxaNcWJY0s7q+nihMODydzvojbwPHCyEy7QTu8fa53UKFdxQKhSIBgeg1GwYpZUwI8XOswVoHZkgplwsh7gQWSSlnAdcIIU4HYkADcLF9b4MQ4i6sLw6AO6WUDbvbJzXoKxQKRQI7GdPfIVLKOcCcbuduT9i/BbhlG/fOAGb0WmdQg75CoVB0YV+3YegXMX1pxPjb4T9n8AuzmX7LM8y64zhe39xC6m0PU/vNfP49/SBee30NJx85kMe/bqBh3Zfsd9hYBpR/yppWHz0lAAAgAElEQVR/v0xlKMZBhT5Sjz6Hl76spGH9CoSmk1Eygqmj8ijU22leupTaLzeyqT1G0DBxaaJDmJVWWohzQCmGL5fGoFUxa3NDkEBrhHAwSizYZomzejJb03V0W5iVKNKyErgJAq2Otb/6VmuBdVuU5dQETq3TbE3rSNp2CrOg03CtQ6TF9gVSiR+CjgRwD4/bnqBrT/PgiNN44Yjr+dWdV/PC1zXc2z6WIUecwVs/PYT5DUFy7niETfPncMOPJvDJLU9T4nWSc+mNzHm3DN3l5bQjB3PWyBxWP/8h5cEoQ1NdDDp2PJtFJvOWbaG1sgwAb/YAsgt9jClMY3CGB0fDRprXVtC0sZnasEHQ6DRbS9WtilmeDI9ltpbhx52Vjp6ejelOxXSlEox1NVtrC8Vot83WwhED05DEokYXcVaPVbM6BFpm0mZryZ77zqOKqCgUCsV3h7if/r6KGvQVCoWiG2rQVygUiu8I2j5eRKVfDPrDS/MJlrVx+G3v0LZlA+mPXM8Zg9I5+9EFFIydSv7cv1IZijH+rmv58XPLcKamc/1JI9n46LUsfmc9Xl0w4vRRVPiH8snSTwjUluP2Z1EwKJPJxZloGz+n5osy6lbVUxeJYUjIcmkUeBykF6eROrAImTmAxrBJlR3Pr2oOEmwLEw5GiYbaMGPRLuKsrYqnOF1WbN9pxfMdTt2K58cFWrYwS9cELr1rIRWnpuHUtQ5hVmLxlK0EV4guwqzECUui2Vr3iczOxut722xtZ9MFuW6dK6/7M3XXFrLm7P048g9P8vnMm1lzyTmcOSSTi579kpTsAVwyKMYtq+uZdtxg3qzzUPnlh+SMOJjpBxWTteET3v64HEPCqGGZpB11Cq9vambLhiaCjdXoLi/+/EGMKc1iv5wUClM0IouW07y2gtYtAZqjW5utpXodHWZrnuw0tPRsNH8GMbefKBphI0Z71KA10lk8pS1sxfXjRmuGYSJNaW87hViJwizYRox+O2ZriiTp5dU7exv9YtBXKBSKPYVg62p0+xJq0FcoFIpu9IUd+N6CGvQVCoUiAYFVs2JfpV9kK8Smtdww57es/3gWl9xwGf+8dx7Hz/kri//7CrddeQTv/PI5js1LZfmAI9jw+TwGHjyVEwtMvnr+a75sDjE23UPxuWfy5pp6qlZvxIxFSCsawWGj89gv201o2XxqV9RRUdNOc7SzIHpakZ+0wQU4BgzG8OfTGDKoaAmxpTlIfXOIUCBKNNCMEQ5ibMNszVqX7+xSEN3h1HssiN5lXb7W6endvSC6LjqLp3Q3W+upILomRI8x821NZhJP7ymztZ1lWvkiBk06njunzyDr8ZcRmkbqQ9cz48WVHPvSH/hg5mwmn30C626/kYgpOfC2K7hv1gqigWZGTx7OkMAaKmc+x7KWMAM8DoYcN5JA8QTeXFZFw6a1mLEInvQcsgr9TBiUQZHPibN+HYGyNTSua2JLKEZLzMSQiWv0NTyZHlJyUvBkp+PJTkdLz0Z605BuH8GoSSgmaYsYltlaKEZrONZRED0WMe01+rIjrm/GIlsZ+UFyBdF3FM9X8f5tILDyZ0m0/oia6SsUCkUCAnAmWQqxP6IGfYVCoUhgXw/vqEFfoVAoEhH9N3STDGrQVygUigR25FXV3+kXgava5jCXbd6P7/34x/yltJxUXePeqgE4vT5+klPN29UBjr3rDH4xcymxYBsXnTqS9pf+wSf1QYKGZNxRA4lNOJ2Zn22kqXwlDo+P/CFFTB2eg7dmFdWfr6Bqcyub2qNETInPoVHkdZAxKI30oUXoBYMJCA+VrWEqm4JUN4UItkYIh6LEQm0YkRBmD2ZrurMziRs3XXO4nJ0ma7pluuZINFuzhVmdSVxr1qELuiZ07eRtotlah8FaQvWsLklZthZh9WS21hOJ920l7NrGPX35H2f4FS/y1W8PZZTfzdRb3+a231zMo/fNY4DHyVPGaELNdcy4YCyznl3GSSVpbBpxEt98+Bn+wqFcd8xw6l78NyteWEpz1OSgnBQKTj6BhZVtrPimlkBtOULT8eUPZsigDMbmp+Fp2oSxaSVNa8pp2dxKQ6Sr2ZrPoZHpcthJXD+e7DQcGVno/gxMlw/D4SEYkwQiBq3hGK0Ru2JWxKA9YhBJEGfFjdaMWKxDmCXNrhWzrGZ2eU+6C7O6XFNJ250i/v9tR60/omb6CoVCkYAQ4NT7xXx4l1CDvkKhUCSwr4d31KCvUCgU3eivoZtk6Be/YQryfbzw4KO8c1YGM469nqsfuoAH/vwip1x8Fp9Nv54RPhfGBb/m67kfkjd6ClcdWsySf7xLW8xkhM/FiAuP4931TaxfVkU00IyvoJQxo/IYX+gjsuwTtiyuYH0gSmPUintmOnWyc1NJH5yHq3gIRnoB9UGDqtYwG+vbCbSEaW+LEG5tIRpsIxYJYsYiHf2NC7M6xFlOu3CK29vVZM2hodnCrHgc391NoKUJy3DNoWt2LB+cutgqtp8Yx9foKsaKG63F0QTdrvf8Cd9TCxh2ZVLVVr2e54dP5aIvX2Lzwre4xvgUXQh+8sC53P7gXEYedzrOp3/L2kCEw+88i9veWElr1VqGTTqEqbkxlv9nAZ9XtpLu1Bh6whDk2OOZvbya2vWbiQaa8aTnkl2Sx2HDcyjNcCHLVxJavYzGNbXUNoc6hFm6AJ9DI8ul28IsL57sdFLyMtHSssGXjfT4CcZMgjHTiuVHbGFWKEZrKGoJs6JWMw1LmGUaXYunmObWBVR6IllhlmLbCETXXNl2WlLPJ8SJQohVQogyIcTNPVy/TgixQgjxlRDiPSHEoIRrhhBiqd1mdb93V1AzfYVCoUikF102hRA68BBwHLAZWCiEmCWlXJHwsC+AiVLKdiHElcAfgfPta0Ep5bhe6YxNv5jpKxQKxZ7Ciukn15LgEKBMSrlOShkBZgJnJD5ASvm+lLLdPpwPFPfin7MVatBXKBSKBOI2DMk0IEcIsSihXd7t6YqA8oTjzfa5bXEp8GbCscd+3vlCiDN74+/rF4N+W1YRw448nVcmTmNla5hPJl9Fe30l/3f6IJ7/bDNn/2wy1762grbqDRx/yljc857gwzUNlKY4OXRsPo5jfsST8zfSuO5LNIeLvKEjOHH/fHLaK6mbv4SasgYaowZBw1qjX+Cx1uhnjijBOXAEQXcmW9oibGpsp6opSLAtQigQsdfoB3tcoy903Vqj7+xco687HHY8v+eC6LoQHfF8l0PDpWs49c41+s6OuH6n0VriGv0uhmti9wqia9uI+Se7Rr+v+fw/N7A2EOV7T23h+CsuYcbZ93DF7Sew5sQbqVnxCf931WG8fsdsJmV5Mc7+FR+9uRhvZgE/O2Ukodce4bOyRipDMSZkeBh0+tGsbNX45MsqWqvWApCaW8KAgRlMKEwnLVRHuOwrGr7ZSOP6JraEDNpi3Quia6TkePFm+0jJy8SZkYGemYvp8WO4fQSiJqGYacXz7TX6bWFrnX4wFCMWTVyfb2KasuNz1X2NPmxdEH1n1+irmP92EFj/v5JoQJ2UcmJCe2yXX1aIi4CJwP0JpwdJKScCFwJ/EUIM3Z0/Dfpw0BdCeIQQnwshvhRCLBdC/M4+P1gIscBOajwvhHD1VR8UCoViZ4lPlnopkVsBlCQcF9vnur6mEMcCtwGnSynD8fNSygp7uw74ABi/y3+YTV/O9MPA0VLKscA44EQhxCTgPuBBKeUwoBHr54xCoVDsJdi/ppNoSbAQGG5Pdl3ANKDLKhwhxHjgUawBvybhfKYQwm3v5wBTgMQE8C7RZ4O+tGizD512k8DRwEv2+SeBXolTKRQKRW/QmzN9KWUM+DnwNrASeEFKuVwIcacQ4nT7YfcDPuDFbkszRwGLhBBfAu8D93Zb9bNL9OmSTXu50mJgGNaypbVAk/1GwHaSGnZC5HKA7IIiUvqyowqFQmEjbC1MbyGlnAPM6Xbu9oT9Y7dx36fAmF7riE2fJnKllIa9xrQYa+nSyJ2497F4cqSxNcqSu47iw7p2fnnjUVz2u9c4dNr3WXn5dLJcOoW/+Rtvv/whWUPGcsfxw1nyxxfZEopx2AG5jLl0Kp/Va3y1uJJg4xZ8BaXsNzqXw0rSiX39IZUL1lHWFu1IzGU6dQqzvWQOz8VTOhQjfQD1wRjlzUE21rfT0hSivTVMONBGJNCMEQltlcRNNFiLbzWnC03XcDh1q7msbaIgq0sS19GZtNW0uBCLDsFWZyWtrslb2E5FLCGSFmbtLskLV3bt+TdOPZpb593H4hef4bVjdFa3hWm4+A9ccO/7DDrsNPZb+G/mNwQ56aZjuWPuWupWL2TwpMOZNjKDr//1PuXBKD6HxsijBuGYfCavLNtC5ZoKQs21uP1ZZJWUMGV4DsOyPIjNK2hYtp7GVVXU1gVpjBpETJkgzNLwZXpIzU/Fm5uJKzsLPTMPzZ9lCbOiljCr2U7etoUtYVYwEiOcIMyKRU2rYpaUHdWyzFikizALVBJ2TxBfFLGj1h/ZI+IsKWWTEOJ9YDKQIYRw2LP9HpMaCoVC8W2ifWvr0vqevly9kyuEyLD3vViKtJVYsalz7YdNB17rqz4oFArFziJQM/1dpRB40o7ra1gJjNlCiBXATCHE3Vjy43/1YR8UCoVip9mHC2f16eqdr6SU46WUB0opD5BS3mmfXyelPERKOUxKeV7imtRt4fD6mLf/4fzyl4dTfeUD1K1eyFs/PYSnX1nFDy4ex/Vvb6RpwzKOPG0yeYue573FVQzwOBh7+dF4T72MRz9ZT+2qxWgOF7nDRnPmuCIKo7XUfTKf6mW1VIetvLJXFxR5HWQOySBrZCmu0pGEU3MtYVZTkI11AdpbrHh+NNCMEQkSC/dgtpYgzNIcLnSXF4fLjcOlbyXM8rr0HounxIVZTs1utjDLqXUKszqKqCQIszoKqWBdi5utJVM8pb8IswDeWl3PSQvzmHzRj3jm0Olcc+3hnH3PPDZ9NptHf3k4b1zxBGPTPaRdcz///e9i3P4srjh9NMbsf/Dp0mq8umBsupvh501ldSyDdxZX0FKxGgBffikFpRlMHpRJdqyRyOovqF9ZQf2aRraEYl2EWWkOnSyXTmpeKql5flLyMtEz89Az8zC96ZhuP+1Rk2DUpDkcozVi0Nwe7YjrR8JdhVnxrTS3XzylJ2FWTzF/JczaBZKc5e/zM30hxGFAaeI9Usqn+qBPCoVC8a0hSHoNfr8kqUFfCPE0MBRYCsSnCRJQg75Codjn2JfDO8nO9CcCo6WUsi87o1AoFHsD+/CYn/SgvwwoAKr6sC8KhULxrbOvl0tMNpGbA6wQQrwthJgVb33ZsUQOKEnnzfIWqq/+K2fd8l8mXfgD1lxyDj6HxsA/PsFLz75P1pCx3H/6aJb8/kkqQzGOOjCPlNMvZ35rKgsXbKa9vhJfQSmjx+RzxKAMzK8/oOLTMla1RmiLmXh1QY7LQWG2l+z98vAOHY6RWUJNe4wNjUHW1QY6hFnRQHNSwiyHy4vu8iYtzEpsyQiz4ola6CrM2tZP031FmAVwz7x7+Ojf/+b9s3wsaQoRvOEh1n88i4GTT2Xyiud4tybAmTcdw81vrqF62YcMOexofnxgLl/8fQ5rAxEmZHg48OhSnEdN4+VlVVSstsR7bn8W2YNKmToqj1E5KWgVK6hbupr6NY3U1ASoi2xfmOXOy7GEWek5mN502mOSQIIwqyUUpTUUoy0UtYVZVvI2LswyDNMSZEUj2xBmmUm/Ryphu+uoRC78ti87oVAoFHsT/cJzfhdJatCXUv5PCJEPHGyf+jzRDU6hUCj2FUQvlkvcG0nqC00I8X3gc+A84PvAAiHEudu/S6FQKPonKrxjmfsfHJ/dCyFygXfptEjuU5q/XslNt/+IiTc8S+OGZax9+CxuvmklV189mSteX0fDui+58Mafk/vpk8z4vJLSFCcTrjmRj5q9PPRhGTUrF6I5XOQP35/zDiqmKFJF1QcfU/l1TYcwK8floMjrIHtYJtn7D8E1ZH8CqblUbGlnfUN7F2FWJAlhlu72dhit9ZUwKy7GShRmJVbMShRmJU5c+rswC+DoT/OZ+pNLeeKgi7jhN8dz+O3vMOSIM3j6+iN4acJhHJzpIeWaP/HSZU/jSc/lF+ceQPSl+/lgyRZ8Do1xxw1m2PnHsTKazpwFq2nasAwAf+FQioZkcnhpFjnRekLL5lO7bDM1NQEqgtsXZqUWZqNn5iEy8jA9fkuYFTR2IMwythJmbatiVqL4KhlhVk+oOP+OEajwDoDWLZxTz779vigUiu8wfbXIYW8g2UH/LSHE28Bz9vH5dPOHVigUin2C7ayA2xdINpF7oxDiHKxyXQCPSSlf6btuKRQKxbeDAHqxhspeR9IhGinly1LK6+y2Rwf8NsPkozNvp3nzas646hIWn3YmAzxOMu58gllPzSZv9BQeOG0k83/zFFtCMaYeVozz9Gv407trWDK/nGDjFtKKRzBhQiFHDsoguvgdyj9a07FG3+fQGJjioCgvhezRA/AOG0ksayDVgRgbmqw1+s0NQQItISKtDcRCAaKhwFbx/Pgafd3l7dg6XO7O9fkJa/S9Lh23HddPcel2fN+K51vxew2HrnWs0XfqPZRrsyPrWkJsv6M/PRit7cwa/V39ebsn1ugDLHzhWV4fU055MMrKC++mYuEcXr11KsPfup9P6oOce/+5XPnyMmq/mc9+U4/hoqEuFv5pDuXBKJOyvAyffib61B/yfwvLKV++jlBzLZ70XHIHD+KEMQWMzk2BDUup/WINdavqqQjGuhRPSXfqZLk0/Nkp+Af4SCnIxp2Xi55dYBmtpWQSiEnaoiYNwSjNoRiN7RGa2qM0tUcIhiyjtZi9Vj9eSGX7xVPM7Rqo7choTZE8QoikWn9ku4O+EOJje9sqhGhJaK1CiJY900WFQqHYc1gLIZJrST2fECcKIVYJIcqEEDf3cN0thHjevr5ACFGacO0W+/wqIcQJvfH3bTe8I6U83N76e+PFFAqFoj/QW3N4u57IQ1hFpDYDC4UQs7oVOL8UaJRSDhNCTAPuA84XQowGpgH7AwOAd4UQI6SUu/UzLtl1+k8nc06hUCj6Pz2EUrfRkuAQoMyuIxIBZgJndHvMGcCT9v5LwDHCih2dAcyUUoallOuBMvv5dotkY/r7Jx4IIRzAQbv74gqFQrHXsXNFVHKEEIsS2uXdnq0IKE843myf6/Exdu3wZiA7yXt3mu2Gd4QQtwC3At6EGL4AIsBju/viyVI0rIArfvkQP7/1Cu4dHeBnP97EvY9eyJlPLCRQW86115+P9tzdvLGslgPS3Iy9/gJeWRdg2fy11JctweHxUXLAaC6cWEJe0xo2zP2IDSvqqAzF0AXkux0UDfCTNTyTnAOH4hh8AM3ODDY1BCirbWNDTRttTSHCrS1EAs3EIsEOAU0czeFCd7rQXR40p53MdXsTkrhax9ZlJ229LgcuvavRmlOzTNZ0gZ3A7TRf6y7Mik80Ek3XenIITDRaS1aY1f3+RLY1v9mTzoS/+9OvuOu4E7n1hV8w6FdPc9B5P8D/yI08fv/7nFacRu3pN/H29L/hLxzK3dPG0fj4Xby/up5ct8648/aHI37Ax5XtzFuwiabylQhNJ71kFCNH5nBkaTaZbeW0fTGfmi83U1UfpC7SKczy6hppDo18jxP/AB+pBRn4inLRswsR6XkYKZkYzhTa2mO0hg2aQzGaw9EOYVZbKNaZuDUksYiBETMxYrEOo7XtCbOALsKsZFHJ3eQQUiKSf6/qpJQT+7I/vc12Z/pSyj/Y8fz7pZRpdvNLKbOllLfsoT4qFArFHkVIM6mWBBVAScJxsX2ux8fYUZR0LAFsMvfuNDtavTPS3n1RCDGhe9vdF1coFIq9DwnSTK7tmIXAcCHEYCGECysx292WfhYw3d4/F5hnF6yaBUyzV/cMBoZjeaDtFjsSZ10HXA78uYdrEjh6dzugUCgUex29VCRQShkTQvwceBvQgRlSyuVCiDuBRVLKWcC/gKeFEGVAA9YXA/bjXgBWADHgqt1duQM7XrJ5ub2dursvtDusi6TgTs/hrtTFvDj5Xk4u8LHmxBv5/Pw7GHbk6dwyzstrF71GxJQce94oWg+7iL/+4zNqV87HiATJGz2F4ycN5MhB6bS/+DAb31/H6rYIEVOS5dIZnOokb0wumSOK8ew3jljOEKraYqypb+ebqhZaGi1hVrjNEmbF465xNIerQ5zVUTzF7cXhcnYKslydAi2XQyPF1UMRFV3riOE77P3tCbO0jjh9YjGVHRutdRFs9fB+97XopDeeftpbd/Op381t8miC9f/HB9dcyt05l9MWM7n2pT/yvUcX0Fq1lhOu/AnHOTbw2gPzqA0bnDMym9JLL+H1dS3MXFROxfLlRAPN+PJLGTC8iJPHFDIqx4Px2QKqF31D3Tf1HUZrhowbrWnkunVS81PwF/rwFeXiyi9Ezy3CTM0i6vDSHjFoi1jCrJZwjOb2KE1BS5gVDseIhg1LmBUxrMIp8eIpcXFWouGaaXQRZpk9iLB2JMxS8fydQMpkZ/FJPp2cQzfbGinl7Qn7ISwH457u/T3w+17rDMkv2TxPCOG3938thPivEGJ8b3ZEoVAo9hZ6Maa/15Hsks3fSClbhRCHA8di/Rx5pO+6pVAoFN8WEsxYcq0fkuygH/9teAqW2dobgKtvuqRQKBTfIpLeTOTudSRrrVwhhHgUS0p8nxDCzR7002+uqWXpQ5fytxEHs6E9wt+/eYYR976P0+vjn1dNZu0tl/FuTYBTC/2MuOVWbv9kI2ULlmBEgqRkD2DYxGFcOKEI14r3WD57ASs2NlMbjuHSBCVeJ4XDs8gdO4S0EUMQJaOoizlZXd/CisoWKmsDtDYECTfXEg00dxROicdIhaYjNB2H24vu8qC7rWLousubYLBmr9F3abg7DNYceJ0JRmvxNfpCdBRQsfY19I5zWo9r9OPF0LcVKo/H+K39TpO2RPbUGv3eShfc+8f/8bemRVxy7K/5zb3X8flxJ6MLwSXnjWKmcyJfvfFHBhx0Ag+dO4YVV5/P+7XtjPK7GX/lUWwZdDgPPbuUDStqaK1ci8PjI3voGKaMLWTKwAw8lV9RvWAB1V9uYX1LmLpIDMPO6/kcGrluB7npHtKK0/AV5ZBalIueW4T052CmZNIaMQlETWttfjhGfXuE+rYIze0R2kJ2PD/aabRmGNYa/fiafMOO7SdqQRILpwA7vUZfsTNI2IkC9P2NZAfu72Nln0+QUjYBWcCNfdYrhUKh+BbZl2P6yfrptwsh1gIn2E5vH0kp3+nbrikUCsW3RD8d0JMh2dU7vwCeAfLs9h8hxNV92TGFQqH4VpASTCO51g9JNqZ/KXColDIAIIS4D/gM+HtfdUyhUCi+Lfpr6CYZko3pCzpX8GDv7zF3rdSsbLQbLyRgmFxz2QSu+drPps9mc+xFZzBp/eu88MwyBngcfO+uM/hEDOXFOato3rSStOIRFI45hEuPHMpIrYHq2bPY+FE5G9qjGNIyWhuSl0LBQUWkjxuHe/QhhDIGsrE5xKraNkuYVR+kvbmFSHuzJcyKdTVaE5qO5nShOZxozgRhllPH6XbgdHc1XPPaSVyX3inM8rp0nJolxooncZ26ldi19hMM17ROYZawK2bF/4F6Emb1lDjdntFaojBrb03iAvzq2sMY8+uPKT74eK4LzuWZ+RVcfttxDPv3f7n1wXfRnS5uuuxQcub+nTdfW4Mu4MjjSkmfdjUzFlewetF6ar9ZhBmLkF48gkGjcjl1/3wGimZCi95jy4I1bF7XRHU4RtCwqmV5dUGmU6fAo5NW7Cd9UCb+gfk48geiZw/ATM0mYOq0RAzaIgZ17VEag1Ea2iI0B6M0tUcJB6PEIkZHMtdK4nYKszrM1oyuwqxE4klcJczqK3rVhmGvI9mZ/r+BBUKIeJnEM7HW6isUCsW+Rz8d0JMh2UTuA0KID4DD7VM/llJ+0We9UigUim+LXrZh2NvYkZ++B/gpMAz4GvinbfKvUCgU+ySCfTumv6OZ/pNAFPgIOAkYBVzb153qzoh0yT+eXc6fn72Mzcdcw5Pn3snAyafy7Hn78e7on1AdjvHT80ejXXAbv354AZu/+B/O1HRKJ4xnyrgBnDYii+gbf2XN61+xtClEW8wk3akxzOekYFw++YeMxjniIGKZxWxujbKipo3lFc001AZoawoSaq4lGmjpEGbFiZusOWwxltPjw+H14fR4cLkdCYVT9I54viXM6tx6XbpttCYSYvjbN1oTCfH8uDCrezw/kZ6M1npie/H8bbEnC6ck8urZd7Hphgeoeu9+Hsgby1nDs2i67D4ufHgB1cs+ZMr0i/lJcYC3z3uOtYEIZwxKZ/QNlzOvMYWX31tB3aqFxEJtpGQPoGj0cKYdUsLBA3yw+D0qP/qCLUurWR+I0hCx4uFeXcPn0Cjw6GQU+kgr9uMfmI+npARHwUAMXw4Rl5/WoEFLyKA5HKMxGKWuLUx9IEJTe4SgLcyKhGMdZmuxSNQSXdkmftsyWuswYdvJeL5iV5CwD4vfdjToj5ZSjgEQQvyLnfByFkKUAE8B+VjC5seklH8VQmQBzwOlwAbg+1LKxp3vukKhUPQBcRuGfZQdrd6Jxnd2IawTA66XUo4GJgFX2dXdbwbek1IOB96zjxUKhWKv4busyB3brTZuvFauAKSUMm1bN0opq4Aqe79VCLESq6jvGcBR9sOeBD4AbtrVP0ChUCh6l+9wIldKqffGiwghSoHxwAIg3/5CANiCFf7p6Z7Lsap2kS4c/PPYw3lq8EXcd+ub6C4Pz908lTU//QGvb27hzCGZjP7DPdw4dy3L3vuEaKCZkkNP4YfHD+e4oTmkLn+H5S98wLI1DVTbRmv/396dx8dVlg0f/12zTxaSJgrXMjkAACAASURBVGnTvWm60N0CBSlLoaUs1SKIPoKPiPqAiK/66kdBtvf1URFFEUEfQagiiCIghbIIUrZCKbKV0pZC6b4lTZql2TN77uePc2Y6STPNlLaZmeb6fj7nkznLzDkH0jtnrvu+rrsiz8OoyWUMO2kivmknEyk/loZAjA/qWlm9q4Vt1W207Q0QaKoj0tFCJNDeezw/RaE1t8+Jxx6n73Q58Ptc+xVaixdbczsEnz1uPzE+v0ehNbdzXyzf6egez+8tqr5vHH/iv2diO+w/Rr/PeP8B9/btcIf+r//eLdz2+xtYdcqZxIxh3vJHmXbzy+x8+wVGz17IQ187gXX/dRHPVrcy7Rgvs2/4NDUTz+WWv65i56q3iAbbcfkKKJ88i3mzRnJWZQn5VauoffVVqt7cxZamYKLQmschlHmcFLmdDC7yUTymiKKxQykcMxxX+ShMUTld+aW0hbtoDcdo6AzvV2ituT1MOBglEkouuBZLFFbriob3K7TWM55/IDo+/zAbqI3+4SAiBcBjwPeMMa3JjYsxxohIr/OSGWMWAYsARjh8h2fuMqWU6ku8DMNR6oiWRxYRN1aD/6Ax5nF78x4RGWbvHwbUHclrUEqpg2Mw0Uhay6EQkRIReUFENtk/B/VyzEwReUNEPhCRtSJycdK++0Vkm4istpeZ6Zz3iDX6Yj3S3wusN8b8JmlX8szvXwGePFLXoJRSB83QXwXX0hnU0glcZoyZCpwH3CEixUn7rzHGzLSX1emc9EiGd04Fvgy8LyLxi7kBuAX4h4hcDuzAqtWvlFJZwWD6a5KaPge1GGM2Jr3eLSJ1wGCg+eOe9Ig1+saYFaTu/zvrYD7L5YChDz/Ndf/xc4It9dx4y9VMeO5WfvaP9Uw7xsuZd36TR1sGs3jJy7TVbKF0/PGcM388l84YSnHjRrb//WE+XL6Lje1WR+wov5tjRx/DyFPGUfzJ2XSNmcmOtgjVrSHW7m5lfXULzfUddOxtIthST7izlVg40G22rJ6duPHELKvz1pU0a5YTj91pW+Bz43fvS8zyuBx2B64Tl3NfJ65D9i+0JgJOu4haz07cvgqt9dWJ21M2F1qLO/5zl/DZ52/hpvfruH3Jd1mwuIZtK56icNg47vzuacg91/HYM5sp8Tg578ufwHfpjVz/7GY+en0tHfW7yCsdTuGw8cw8YTgXzxzBqPBu2l77F7te/Yjt25rZFYgkCq2VeJwM9bko87oYVFlM0dgyisaNwDV8LI7Bo4kWltMac9ISilLfEaahM0JLKEJ9a4i9HVZnbjgUtZKy7Nmyenbi9lZoDXokX8VimozVHwwHM3NWmYisTFpfZPdHpiOtQS1xInIS1jS1W5I23ywiP8L+pmCMCfV10iPekauUUrnloDpyG4wxs1LtFJEXgaG97Lqx2xkPMKjF/pxhwF+BrxiTGFp0PdYfCw/WoJdrgZ/2dcHa6CulVDJjDrmTdt9Hmfmp9onIHhEZZoypOdCgFhE5BngGuNEY82bSZ8e/JYRE5D7g6nSuqd8mN1dKqdxgetQ/Sr0coj4HtYiIB1gCPGCMWdxjX3wUpGCVu1+Xzklz4km/bOoETvveYly+fE67cAHXF2/gju8vxu8ULv7xp9g442Ju+vVy9ry/nILyCmbMPY7vz6mkcPVT1K14jY8e/5A1LUHCXYbhPhfTyvyMOnU0Q04/CZn4SXbH8lhT28qOpk5W7WiisbaNtr2tBJpriXS2EgvtH893uD37xfPdXg9urwuPd98EKn6fC4/LQWGPeL7f48TncnafOMVOynLYxdbiSVk9J05JN56fXHzt406ckkom4/kAr85t5v+espQffu8Uflu0kBU330blnAv48mcmc8aWx7jnZ8/TEuni0rPHUnHDTfzu3Vr+9dxH7N26Bnd+EUOnnsiwsYP46sljmF4YJvzS02xf+i671taxpSNMS8T6Bl3kdjLc52JYqZ+CIfmUjC+laNwIvKPG4hpeSbRoKAGHj+bOGPUdEeo6wtR3hGjpjFDXFqKxPUQwECEcsBKzrLh+jGg4RMwu4JdIzEokZe1LzAK6FVqL04lTjqD46J0jr9dBLSIyC7jKGHOFvW0OUCoiX7Xf91V7pM6DIjIY65/1aqyKyH3KiUZfKaX6jzmYjtyPfxZjGullUIsxZiVwhf36b8DfUrx/3sc5rzb6SimVzNBfQzYzQht9pZTq5uguw5ATjf6He4I4tq3hvruu4aKyNh6acSW7gxG+ddWJhL/6M77+P/9m6+vP4S0sYfKZp/GzhVOobHiXDX98kN3v1vJmfQctkS5KPE6mF3kZM2c0I+adhGvGHBp8Q3h/dzsrdzSxo7GDmupWWho66WysJtzW1K3QWvL4fIfL0+vEKV6/C4/fjdfvwut1UWjH9HubOMXnsoqseeNj9JPG6fecOKXnWP1U8fy4A8Xzk/UVz++9mFtm4/kA//+Ma/jK3DFsuuoObv76ryibeCJP3DCXCY2rWHzm/7C+LcQXpg/hhN/8iEfrC/jj4++y5/3lOFwehkw5lbmnV3D6uFLmjjmGrtf+zs5/rWDXiio+bA0lJk4pcjsY7nMxqshL6fhB5JfnUzxxFPmVlbhHTyR2zFBC3iL2dkZpDESoaQ+xpz1EbXOQtlCUxvYQbR1hQoEooWCESHDfxCnJ4/OT4/nWeP1DmzhF4/mH6DCO3slGOdHoK6VU/9EnfaWUGjj6b/RORmijr5RSSQwG0w+jdzJFG32llEqmT/qZF2pr5te/+DbzXrmNpbe+wJt7A1x18RTKf/UXFv7hLdY9/ywOt4eJZ8zjJ5+bzgnRLWy96y7efXYz2zoi1IdiFLkdfKLIS+Wc0Yw+90S8J55Dc9FY1tV28Ob2vbyzpZHO1hBNe9rpqN+5XycukOjEdfnycbg8ePKLcOcX4cnLx+tz4/G7EslZXq+LAp+LAp/bSs5KrFszZ3ntGbPiCVm++LozXnDNkSi4lkjIInUnblzybFnQeydub7NlHe4ia0favIpiiv7+NJ/+6h34BpXz4E8voPDua3juT2+wrL6TC8YUcdo91/Kicwq/eGAlO958EdMVY8jUU5l92hi+MXsM4wZ5cax8kp1PLWXbi9tY0xxkT8iaLavA5WC4z01FvoeSCSWUHFtO/rBSCiaMx10xmVjxCEL5g9kbiNHYGaWmLURdh9WJW9MSoDMco6U9TLAjQihgdeKGQ1EioTCxUIBYOJDoxE102monbnYwBhMJ931cjsqJRl8ppfpP/yRnZYo2+kop1dNR/I1JG32llEpmzFEdJsuJRn/oiHIu33gfP796CS2RLr5xwUTG3/c4Cxe9w8olT2JiMY6dt4AfXzKTuZ7dbPvNr3n7kXWsag4SiBk7nu/j2NNHUbnwk/hPWUhL6UTW7Ongta2NvLGpgfqqVoKdYTrqqwi1NBDuaCEWDiSuwenxJ+L5Ln8BTpcHl6+gWzzfaydl+fxuCnwuivM8FHhdeF0OK5bvcSbi+dbkKdYEKvti+1Ys3yHSLZ7vdLAvQYve4/kO6R7PT07WykQ8/0iH/if8+1VO+tqdiMPJA7+8jEmP/YTf/+JF6kMxFg4rZN79P+SNIWdw3Z/fYfOKF4iFAwyZciqzzzyWH8ydwDRHPV3vvceuxU+w+V+bWFPf2SOe72JcgYfBU8sYPG0YZTPG4y4tw1Mxia7SMUQKh9LYGaWhM0pVa5Da9hDVewPUtASpaw0RDscIdlrx/HAgmjKer0lZ2UlH7yil1EBhDCamjb5SSg0Ixhi6ItFMX8YRo42+UkolM+iTfqYNCTZw01V/Z1y+h1POGcu4+x9nwR/e4p3HnsDEYkye/yl+funxzPdUse3WW3jj4fd5pylIzFjx/OOLfUw6cwyVCz9J3pwLaS6dyHu1HbyyuYHXN9RTX9VKc80ewp0tacXzPXlFOL3+tOL58YJryePzk+P5yePz42PzHXL44/npTpqSC/F8gFmX3o7D7eHR313JpId/xG9veh6nwPkjj+Hsh/8fy4fM5ep732bjsueIhQOUT5/DafMm8cOzJjBN9tD+9F9oWLuZTf/cwJr6TnYFIt3i+RMLvd3i+b6J03AOGkJXWQWRwqHUd0ap64hQ1Rqkui1I9d4AVU2d1LWG6GgLE43E0ornJyZE13h+VtFGXymlBghjDF1aT18ppQaOo3n0jk6MrpRSyezRO+ksh0JESkTkBRHZZP8clOK4mIistpenkraPFZG3RGSziDxiT6LeJ230lVIqSXz0TjrLIboOeMkYMwF4yV7vTcAYM9NePpO0/ZfA7caY8UATcHk6J82J8M7uXU2cOLiEzy39DXsqTuesX69g7TNP4PYXMH3hudzxn8dxfPsaNvz3bax4ehNrWoIATCzwMjrPxbHnVFJx/ul4Zn+a+sIKVla1sXxzA29trKeuqpXW2lo6G6uJhYOEO1p6nSnL5cu3i6tZRdacLgdenxtfvrvbTFnFeW4KfO5EJ25BfOYst5M8uyM3PlNWb524TsfBz5TVW8cu9H8nbn/WYvMPGspLd1yC66YruPXudyj3urjs+vmUX3Qxj0Ym8JO73mD7v5cCMPyEczl7/ni+f0YlEwJb2bvkL2x8bCVNW5tZ1RRIJGUVuR2M8rsZN8jH4ClllE0fSdmMcXgqp+IYNYkubyHB/MHUd0So64iwsyVIjd2JW9MSoKY5SLAjQrDT6shNVWQtGg5gYjHtxM1iXf3TkXsBcKb9+i/AK8C16bxRrH/I84D/THr/j4E/9PVefdJXSqlk9pDNNMM7ZSKyMmm58iDOVG6MqbFf1wLlKY7z2Z/9pohcaG8rBZqNMfGvG1XAiHROmhNP+kop1W8OLiO3wRgzK9VOEXkRGNrLrhu7n9IYETEpPmaMMaZaRCqBl0XkfaAl3QvsSRt9pZRKYjh8o3eMMfNT7RORPSIyzBhTIyLDgLoUn1Ft/9wqIq8AxwGPAcUi4rKf9kcC1elcU040+oPy3Fy07lm+9nwDb//5BbateIrCYeM45cJ5/O6iaQxfu4R3f3U/K16vYmN7GL9TmHaMl+knDaf02CGMWDAP5/HnsMtRylvbm3l5Qz0fbN1LQ3UrrbVVBJpqCbU1JQpfgRXPT07K8uQX4c4rwpNfiMfvxul04Mt34/W78fhc+H29x/P9HiduhxW/j8fzfS4rpm/F9vfF87vF8NOI58dj6OnE86VHwD2X4/kAm++7jPfOPYcHlu/k5BI/X7jrMraf8S3u+6CWe/76MnveX44nv4jRs87g4gUTuXzWSMp3vs7uRx9hw5K1rN3ZQlMkRn0ohlNgsNfJKL+bsUPzGTyljMEzKhg0ZRye8TNg6DiixSPpjBoa26PsbgtR3RqkujVI1d4AtS0BGlpDBDsjBDvChAJRYrEuIqEokWAwEc+PRa1krH1F1iLd4vYHiuenittrPP8IMIaucL+UYXgK+Apwi/3zyZ4H2CN6Oo0xIREpA04FfmV/M1gGfB54ONX7e6MxfaWUSmagq6srreUQ3QKcLSKbgPn2OiIyS0T+ZB8zGVgpImuAZcAtxpgP7X3XAt8Xkc1YMf570zlpTjzpK6VUfzH0T5VNY0wjcFYv21cCV9iv/w1MT/H+rcBJB3tebfSVUiqZIRFmOxrlRKPvnTCR2XduYN2zS+iKhhl23Hyu/NIsrj55GJ0P3Mzy373A8m3N1IdiDPY6OXGQnwnnVTJm4Rw8FZOJTZrD+pYulu9oYNn6OrZva2Lvnnba92xLFFjrOQG60+vH5fHjzj8Gt68gMQG6L8+D1+/CYY/T9/pdFOa5KfS5KPJ7KLTj+AU+F/keFz6XNSmK12XH9XvE8ZMnQI+PzXdgx+/tuP6BxuZDj8JrSf/dDiWen62x/LjFI4/j9cYAl5wwjDkP386DrSP52c0v07DlA9pqtlA4bByT5szmOwuO5cIJxbD8QTY98k82PbeV1UkToHscQrnXxdh8NyMri63x+TPGUTh5Mp7KqURLxhDKK6W+I0ogYhIF1qqaAlQ1BahrDdLcFkqMz48EY4SCEUyXIRLsJBbaNzb/QBOmgNXQ6Nj8bGC0DMPHISJ/FpE6EVmXtC2ttGOllMqYgxunn3OOZEfu/cB5Pbalm3aslFIZYYwhFo6mteSiI9boG2OWA3t7bL4AK10Y++eFKKVUVjF2+K3vJRf1d0w/3bRj7HTmKwFGjBzVa0qbUkoddjpz1pHRR9oxxphFwCIA96DRpu6pRxj6ibmMmjQiUWBt47ev5rUnNiYKrE0u9HLc5FLGn/8JBp+7gK7JZ9AQc7FyZ3uvBdZCbU3EwoFEp9iBCqxZM2PZs2T53Lg8jpQF1gp8Lnt2LKvAmlVULXWBtXgSVqLTto+ErN46cOHoLrDWU3Ugyo9uWoD3u7dx0SNrWfHk32it2ojT42fEiZ/qXmDtnl+z8bGVrF1Xz5aOMO3RLjwOocAlKQusOUdOJDJoNE0xF40t1gxZLaFoygJroUDUSsYKRYkEOzGxWMoCa/GkrOQOXEivwJp24PYDAyaWsmnKef3d6KeVdqyUUpliMP1VZTMj+jsjN552DAeRNqyUUv3GgOkyaS256Ig96YvIQ1i1ostEpAr4b6w043+IyOXADuALR+r8Sin1cRgDsfDRG0Y7Yo2+MeaLKXbtl3bc52fFopxx+X9x5xdmMNbdSfO9P+a53y5j+Z52WiJdDPW5OLEsj/ELxjP6M2fhmnUeNZ5y3tnWyo7mAMvW11G1o5m9NU101O8k1NJAJNC+32QpDrcHty8fl78gEcv3+P12PN+VmCwlz+/G43JQnOfZr7haPCErubiaQ6w4vhXTt2L5DumekHU4i6vBgWP5ye9Jlgux/LirNzzB3bvyuO0Hz7L73aW4/AVUzrmAoRXFXH3eJM4Z7iS27F4+fOQFNry8g3WtIWqD1hC7Eo9VXG2w10l5ZTFDppdTNmMc+RMn4Rk3neigkbR5imkIxKwYfluQ3a1BWjoj1LQEqWkO0GonZIWCkX3xfLu4WpddWC3Wrbja/glZOllKljJGY/pKKTWQdGmjr5RSA4QO2VRKqYHDAF052kmbDm30lVIqmTHakZtpEyqGsHRelI3XXsryt2t4dcte6kMxSjxOzi3PZ8JZFVReeAae2Z+mobCClbvbWb55B29trKejNURjTRsd9TsJNu3pVlGzZzKWw+WxZsiyK2p6fW58+W48fjcerxOf351IxvK4HBR69yVj+d1O8tzORAducjJWvCM3VUVNpyN1By7QbRvs34HbbdtR3oEbN/22LWz/91IAhp9wbiIZq+IYN7z2d7be9jSbn93CqqZAoqJmkdvRLRkrvzyf0qljOWbKJNyV0+gqHUNH/mDqO6PUNdozY7XuS8ZqC0b3q6gZDkWJhMKJ2bGSk7G0Azc3GU3OUkqpAUQbfaWUGkg0I1cppQaOfsrITWd+ERGZKyKrk5agiFxo77tfRLYl7ZuZznlz4klfdm7l9yd/g/VtIQCG+lycP/KY/ZOxqltZ9tYWVm9upGF3K621uwl3tqRMxnL7C3AlJWM5vf6UyViFPhdFSclYHpcjZTKW22nF8r12MpbTQUaTsXK5sFoqO99ZxpiTz+Gz50zgqpNHM7L+PWrvvoZNH+1KmYxVOSSPIVPKKJs2itLp43EMGrJ/MlZtZyIZq2pvgNqWAHuagwQ7I0TDsZTJWFE7nh9PxrIWjeXnIkO/jdOPzy9yi4hcZ69f2+1ajFkGzATrjwSwGXg+6ZBrjDGLD+akOdHoK6VUvzGGrv4ZvXMBVqkasOYXeYUejX4Pnwf+ZYzpPJSTanhHKaWSGGM96aezHKK05xexXQI81GPbzSKyVkRuFxFvOifVJ32llOrhIGbFKhORlUnri+y5QAAQkReh1zmgbux2vj7mF7FL0U8HliZtvh7rj4UHa+6Ra4Gf9nXBOdHo17eEaPHFuGBMESUTShh3/vEMmn8+obEns64+wCsbGlm2fi27dzbTVNvcrahaPKYK4HB5cHr9KYuquTwOvD57ohS/mwKfa7+iagU+Fz6X0yqg5rRi+W5nfGx+96JqTofgwIrXOx3sey19x/Ghx1h9e1uqOH7Pfcnv6SmdWH42xvGTLf7TdZw1VIi+9ACbvvkSb75ixfHbo10EYga/U6jIczO+wMOwCSUMmV5OydSxFEyagnvsVKIlo+nyFlITgsZAlJ172qhuDbK7OUBVU4C61uB+RdW6ol2EQ1Fi4UBiXH5fRdWAxJh90Dh+TjAH9RTfYIyZlfqjzPxU+0TkYOYX+QKwxBgTSfrs+LeEkIjcB1ydzgVreEcppZLZ4/TTWQ7Rwcwv8kV6hHbsPxSI9fR3IbAunZPmxJO+Ukr1F0O/FVzrdX4REZkFXGWMucJerwBGAa/2eP+DIjIY60v9auCqdE6qjb5SSiUzhlj4yDf6xphGeplfxBizErgiaX07MKKX4+Z9nPNqo6+UUkmMgS6jZRgyauiQAq598EZk5tkE/KWs3dPJ8m2NLHt5JfVVrTTVNtJRt5Nwe9N+SVjicOLyF+D25ePOL8LtK8CdX4QvPy+RfOX1u/H4XDhdDory3BT63BTZCVl+jzPReRsvqOZ2SCIBy0rGkv06bzUJ68gacs2lPPJGFevbwuwNx3CKlYQ13Ofm2EIPZRNLGDJ9KGUzxuOfOBXXmMnEBo2kzVlAQyBK7d4wLSGr87a6aV/nbVtbiGBnhHAgahdT25eEZbpi3TpvrY5bTcI6GsW00VdKqYHBAEdxvTVt9JVSqid90ldKqQGiy0BYZ87KrI7SEfyfhpl8uGgDna2hA8bwnR4/nvwiXL58PPlFVmG1FDH8gjy3nXTlptjvThRR6y2G7+uRhOW0J0bpK4bvTJrcRGP4h8+fn9lEicfJuHw388aXMHhqGYNnVJA3ZNB+MfztgSi1bWGqtwWpbt2dKKTWFoweMIafiN+nUUgtVdxeY/i5ScM7Sik1QBiMhneUUmqg0I5cpZQaYLTRz7AdO/fwt1vv6hYLdXr8uLx+/IPKuxVP8/q9ePzWhOZenxunS/ClKJ7m9zjJdzvxuqzYvVPA63ImJjTvOf4+Hq932sHwA01ofijF0zR237df3HsZvonTcI48luigUbQYLw2BGNXhGDtbAtTUhqj+sIGqpp3UtYboaAsTCkYIdkSsuH0oSiwa7TaheW/j7+P9RTr+fuAwRkfvKKXUgGHQ0TtKKTVgaExfKaUGGA3vKKXUAGHF9DN9FUdOTjT6Ln8BY09baM1u5XbsS7LyuijOc1PQW4E0pwOvPcOVlVDl6LODNt0Cacmds6DJVZlwlfN86t4LEVyxl2BnLaFAlHAgQizWRTQcSXTQRu1OWhOLJTpou6KRRCerdtCq3uiTvlJKDRAG6JcpVDJEG32llEpiMDp6RymlBgpr9I42+hk1dXQxr//y3Exfhsoii2//Q6YvQR2tjvKOXEffhxx+InKeiGwQkc0icl0mrkEppXoTf9JPZzkUIvIfIvKBiHTZk6GnOq7X9lJExorIW/b2R0TEk855+73RFxEncCewAJgCfFFEpvT3dSilVCoxk95yiNYBFwHLUx3QR3v5S+B2Y8x4oAm4PJ2TZuJJ/yRgszFmqzEmDDwMXJCB61BKqf10YZVhSGc5FMaY9caYDX0c1mt7KdZY8HnAYvu4vwAXpnPeTMT0RwC7ktargE/2PEhErgSutFdDeX7/un64tv5SBjRk+iIOo6PtfuDou6eBdD9jDuWDGwgvvYcdZWke7hORlUnri4wxiw7l/D2kai9LgWZjTDRp+4h0PjBrO3Lt/3CLAERkpTEmZcwr1+j9ZL+j7Z70ftJnjDnvcH2WiLwIDO1l143GmCcP13kORiYa/WpgVNL6SHubUkodVYwx8w/xI1K1l41AsYi47Kf9tNvRTMT03wEm2D3PHuAS4KkMXIdSSmW7XttLY4wBlgGft4/7CpDWN4d+b/Ttv0rfBpYC64F/GGM+6ONthzNGlg30frLf0XZPej9ZRkQ+KyJVwGzgGRFZam8fLiLPQp/t5bXA90VkM1aM/960zmuO4swzpZRS3WUkOUsppVRmaKOvlFIDSFY3+rlarkFE/iwidSKyLmlbiYi8ICKb7J+D7O0iIr+z73GtiByfuSvvnYiMEpFlIvKhnTb+XXt7Tt6TiPhE5G0RWWPfz0/s7b2mtYuI117fbO+vyOT1pyIiThF5T0T+aa/n+v1sF5H3RWR1fCx8rv7OZZOsbfRzvFzD/UDPsb7XAS8ZYyYAL9nrYN3fBHu5EsjGSmJR4AfGmCnAycC37P8XuXpPIWCeMeYTwEzgPBE5mdRp7ZcDTfb22+3jstF3sTr74nL9fgDmGmNmJo3Jz9XfuexhjMnKBatHe2nS+vXA9Zm+roO4/gpgXdL6BmCY/XoYsMF+fQ/wxd6Oy9YFa2jY2UfDPQF5wCqsLMcGwGVvT/z+YY2cmG2/dtnHSaavvcd9jMRqBOcB/8SamC1n78e+tu1AWY9tOf87l+kla5/06T39OK004yxVboypsV/XAuX265y6TzsUcBzwFjl8T3YoZDVQB7wAbCF1Wnvifuz9LVhD5LLJHcAP2Tfp04HS9HPhfsAqePm8iLxrl2WBHP6dyxZZW4bhaGaMMSKSc2NlRaQAeAz4njGmVZIm6c21ezLGxICZIlIMLAEmZfiSPjYRWQjUGWPeFZEzM309h9FpxphqERkCvCAiHyXvzLXfuWyRzU/6R1u5hj0iMgzA/llnb8+J+xQRN1aD/6Ax5nF7c07fE4Axphkrs3E2dlq7vSv5mhP3Y+8vwkqDzxanAp8Rke1YVRjnAb8ld+8HAGNMtf2zDusP80kcBb9zmZbNjf7RVq7hKaxUaeieMv0UcJk9+uBkoCXp62tWEOuR/l5gvTHmN0m7cvKeRGSw/YSPiPix+ifWkzqtPfk+Pw+8bOzAcTYwxlxvjBlpjKnA+nfysjHmS+To/QCISL6IOYcM9AAAAmhJREFUFMZfA+dg1Z/Pyd+5rJLpToUDLcCngI1Y8dYbM309B3HdDwE1QAQrtng5Vsz0JWAT8CJQYh8rWKOUtgDvA7Myff293M9pWPHVtcBqe/lUrt4TMAN4z76fdcCP7O2VwNvAZuBRwGtv99nrm+39lZm+hwPc25nAP3P9fuxrX2MvH8T//efq71w2LVqGQSmlBpBsDu8opZQ6zLTRV0qpAUQbfaWUGkC00VdKqQFEG32llBpAtNFXGSciMbuS4gd25csfiMjH/t0UkRuSXldIUrVTpQY6bfRVNggYq5LiVKxEqQXAfx/C593Q9yFKDUza6KusYqyU+yuBb9vZlU4RuVVE3rHrpH8DQETOFJHlIvKMWHMu3C0iDhG5BfDb3xwetD/WKSJ/tL9JPG9n4So1IGmjr7KOMWYr4ASGYGUztxhjTgROBL4uImPtQ08CvoM138I44CJjzHXs++bwJfu4CcCd9jeJZuBz/Xc3SmUXbfRVtjsHq6bKaqxyzqVYjTjA28aYrcaqmPkQVrmI3mwzxqy2X7+LNdeBUgOSllZWWUdEKoEYVgVFAb5jjFna45gzseoBJUtVUySU9DoGaHhHDVj6pK+yiogMBu4Gfm+swlBLgW/apZ0RkYl21UWAk+wqrA7gYmCFvT0SP14p1Z0+6ats4LfDN26s+Xj/CsRLOP8JKxyzyi7xXA9caO97B/g9MB6rjPASe/siYK2IrAJu7I8bUCpXaJVNlZPs8M7VxpiFmb4WpXKJhneUUmoA0Sd9pZQaQPRJXymlBhBt9JVSagDRRl8ppQYQbfSVUmoA0UZfKaUGkP8FDDDoBdR2Fq0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_encoding = positional_encoding(50, 512)\n",
    "print(pos_encoding.shape)\n",
    "\n",
    "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
    "plt.xlabel('Depth')\n",
    "plt.xlim((0, 512))\n",
    "plt.ylabel('Position')\n",
    "plt.colorbar()\n",
    "plt.show() # 在这里左右边分别为原来2i 和 2i+1的特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.掩码\n",
    "为了避免输入中padding的token对句子语义的影响，需要将padding位mark掉，原来为0的padding项的mark输出为1\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=311819, shape=(2, 1, 1, 5), dtype=float32, numpy=\n",
       "array([[[[0., 0., 1., 1., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0., 1., 1.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def create_padding_mark(seq):\n",
    "    # 获取为0的padding项\n",
    "    seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
    "    \n",
    "    # 扩充维度以便用于attention矩阵\n",
    "    return seq[:, np.newaxis, np.newaxis, :] # (batch_size,1,1,seq_len)\n",
    "\n",
    "# mark 测试\n",
    "create_padding_mark([[1,2,0,0,3],[3,4,5,0,0]])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "look-ahead mask 用于对未预测的token进行掩码\n",
    "这意味着要预测第三个单词，只会使用第一个和第二个单词。 要预测第四个单词，仅使用第一个，第二个和第三个单词，依此类推。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_look_ahead_mark(size):\n",
    "    # 1 - 对角线和取下三角的全部对角线（-1->全部）\n",
    "    # 这样就可以构造出每个时刻未预测token的掩码\n",
    "    mark = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
    "    return mark  # (seq_len, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0. 1. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 0.]], shape=(3, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# x = tf.random.uniform((1,3))\n",
    "temp = create_look_ahead_mark(3)\n",
    "print(temp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.Scaled dot product attention\n",
    "![](https://www.tensorflow.org/images/tutorials/transformer/scaled_attention.png)\n",
    "进行attention计算的时候有3个输入 Q (query), K (key), V (value)。计算公式如下：\n",
    "$$\\Large{Attention(Q, K, V) = softmax_k(\\frac{QK^T}{\\sqrt{d_k}}) V} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "点积注意力通过深度d_k的平方根进行缩放,因为较大的深度会使点积变大，由于使用softmax，会使梯度变小。\n",
    "例如，考虑Q和K的均值为0且方差为1.它们的矩阵乘法的均值为0，方差为dk。我们使用dk的根用于缩放（而不是任何其他数字），因为Q和K的matmul应该具有0的均值和1的方差。\n",
    "\n",
    "在这里我们将被掩码的token乘以-1e9(表示负无穷),这样softmax之后就为0,不对其他token产生影响。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scaled_dot_product_attention(q, k, v, mask):\n",
    "    # query key 相乘获取匹配关系\n",
    "    matmul_qk = tf.matmul(q, k, transpose_b=True)\n",
    "    \n",
    "    # 使用dk进行缩放\n",
    "    dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
    "    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
    "    \n",
    "    # 掩码\n",
    "    if mask is not None:\n",
    "        scaled_attention_logits += (mask * -1e9)\n",
    "        \n",
    "    # 通过softmax获取attention权重\n",
    "    attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)\n",
    "    \n",
    "    # attention 乘上value\n",
    "    output = tf.matmul(attention_weights, v) # （.., seq_len_v, depth）\n",
    "    \n",
    "    return output, attention_weights\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用attention获取需要关注的语义"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_out(q, k, v):\n",
    "    temp_out, temp_att = scaled_dot_product_attention(\n",
    "    q, k, v, None)\n",
    "    print('attention weight:')\n",
    "    print(temp_att)\n",
    "    print('output:')\n",
    "    print(temp_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "attention测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention weight:\n",
      "tf.Tensor([[0. 1. 0. 0.]], shape=(1, 4), dtype=float32)\n",
      "output:\n",
      "tf.Tensor([[10.  0.]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 显示为numpy类型\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "temp_k = tf.constant([[10,0,0],\n",
    "                      [0,10,0],\n",
    "                      [0,0,10],\n",
    "                      [0,0,10]], dtype=tf.float32)  # (4, 3)\n",
    "\n",
    "temp_v = tf.constant([[   1,0],\n",
    "                      [  10,0],\n",
    "                      [ 100,5],\n",
    "                      [1000,6]], dtype=tf.float32)  # (4, 3)\n",
    "# 关注第2个key, 返回对应的value\n",
    "temp_q = tf.constant([[0,10,0]], dtype=tf.float32)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention weight:\n",
      "tf.Tensor([[0.  0.  0.5 0.5]], shape=(1, 4), dtype=float32)\n",
      "output:\n",
      "tf.Tensor([[550.    5.5]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 关注重复的key(第3、4个), 返回对应的value（平均）\n",
    "temp_q = tf.constant([[0,0,10]], dtype=tf.float32)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention weight:\n",
      "tf.Tensor([[0.5 0.5 0.  0. ]], shape=(1, 4), dtype=float32)\n",
      "output:\n",
      "tf.Tensor([[5.5 0. ]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 关注第1、2个key, 返回对应的value（平均）\n",
    "temp_q = tf.constant([[10,10,0]], dtype=tf.float32)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "attention weight:\n",
      "tf.Tensor(\n",
      "[[0.  0.  0.5 0.5]\n",
      " [0.  1.  0.  0. ]\n",
      " [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)\n",
      "output:\n",
      "tf.Tensor(\n",
      "[[550.    5.5]\n",
      " [ 10.    0. ]\n",
      " [  5.5   0. ]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 依次放入每个query\n",
    "temp_q = tf.constant([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=tf.float32)  # (3, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.Mutil-Head Attention\n",
    "![](https://www.tensorflow.org/images/tutorials/transformer/multi_head_attention.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mutil-head attention包含3部分：\n",
    "- 线性层与分头\n",
    "- 缩放点积注意力\n",
    "- 头连接\n",
    "- 末尾线性层\n",
    "\n",
    "每个多头注意块有三个输入; Q（查询），K（密钥），V（值）。 它们通过第一层线性层并分成多个头。\n",
    "\n",
    "注意:点积注意力时需要使用mask， 多头输出需要使用tf.transpose调整各维度。\n",
    "\n",
    "Q，K和V不是一个单独的注意头，而是分成多个头，因为它允许模型共同参与来自不同表征空间的不同信息。 在拆分之后，每个头部具有降低的维度，总计算成本与具有全维度的单个头部注意力相同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构造mutil head attention层\n",
    "class MutilHeadAttention(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super(MutilHeadAttention, self).__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "        \n",
    "        # d_model 必须可以正确分为各个头\n",
    "        assert d_model % num_heads == 0\n",
    "        # 分头后的维度\n",
    "        self.depth = d_model // num_heads\n",
    "        \n",
    "        self.wq = tf.keras.layers.Dense(d_model)\n",
    "        self.wk = tf.keras.layers.Dense(d_model)\n",
    "        self.wv = tf.keras.layers.Dense(d_model)\n",
    "        \n",
    "        self.dense = tf.keras.layers.Dense(d_model)\n",
    "        \n",
    "    def split_heads(self, x, batch_size):\n",
    "        # 分头, 将头个数的维度 放到 seq_len 前面\n",
    "        x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n",
    "        return tf.transpose(x, perm=[0, 2, 1, 3])\n",
    "    \n",
    "    def call(self, v, k, q, mask):\n",
    "        batch_size = tf.shape(q)[0]\n",
    "        \n",
    "        # 分头前的前向网络，获取q、k、v语义\n",
    "        q = self.wq(q)  # (batch_size, seq_len, d_model)\n",
    "        k = self.wk(k)\n",
    "        v = self.wv(v)\n",
    "        \n",
    "        # 分头\n",
    "        q = self.split_heads(q, batch_size) # (batch_size, num_heads, seq_len_q, depth)\n",
    "        k = self.split_heads(k, batch_size)\n",
    "        v = self.split_heads(v, batch_size)\n",
    "        # scaled_attention.shape == (batch_size, num_heads, seq_len_v, depth)\n",
    "        # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)\n",
    "        \n",
    "        # 通过缩放点积注意力层\n",
    "        scaled_attention, attention_weights = scaled_dot_product_attention(\n",
    "        q, k, v, mask)\n",
    "        # 把多头维度后移\n",
    "        scaled_attention = tf.transpose(scaled_attention, [0, 2, 1, 3]) # (batch_size, seq_len_v, num_heads, depth)\n",
    "\n",
    "        # 合并多头\n",
    "        concat_attention = tf.reshape(scaled_attention, \n",
    "                                      (batch_size, -1, self.d_model))\n",
    "        \n",
    "        # 全连接重塑\n",
    "        output = self.dense(concat_attention)\n",
    "        return output, attention_weights\n",
    "        \n",
    "        \n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试多头attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 60, 512) (1, 8, 60, 60)\n"
     ]
    }
   ],
   "source": [
    "temp_mha = MutilHeadAttention(d_model=512, num_heads=8)\n",
    "y = tf.random.uniform((1, 60, 512))\n",
    "output, att = temp_mha(y, k=y, q=y, mask=None)\n",
    "print(output.shape, att.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "point wise前向网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def point_wise_feed_forward_network(d_model, diff):\n",
    "    return tf.keras.Sequential([\n",
    "        tf.keras.layers.Dense(diff, activation='relu'),\n",
    "        tf.keras.layers.Dense(d_model)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_fnn = point_wise_feed_forward_network(512, 2048)\n",
    "sample_fnn(tf.random.uniform((64, 50, 512))).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.编码器和解码器\n",
    "\n",
    "![](https://www.tensorflow.org/images/tutorials/transformer/transformer.png)\n",
    "\n",
    "- 通过N个编码器层，为序列中的每个字/令牌生成输出。\n",
    "- 解码器连接编码器的输出和它自己的输入（自我注意）以预测下一个字。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 编码层\n",
    "每个编码层包含以下子层\n",
    "- Multi-head attention（带掩码）\n",
    "- Point wise feed forward networks\n",
    "\n",
    "每个子层中都有残差连接，并最后通过一个正则化层。残差连接有助于避免深度网络中的梯度消失问题。\n",
    "每个子层输出是LayerNorm(x + Sublayer(x))，规范化是在d_model维的向量上。Transformer一共有n个编码层。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNormalization(tf.keras.layers.Layer):\n",
    "    def __init__(self, epsilon=1e-6, **kwargs):\n",
    "        self.eps = epsilon\n",
    "        super(LayerNormalization, self).__init__(**kwargs)\n",
    "    def build(self, input_shape):\n",
    "        self.gamma = self.add_weight(name='gamma', shape=input_shape[-1:],\n",
    "                                     initializer=tf.ones_initializer(), trainable=True)\n",
    "        self.beta = self.add_weight(name='beta', shape=input_shape[-1:],\n",
    "                                    initializer=tf.zeros_initializer(), trainable=True)\n",
    "        super(LayerNormalization, self).build(input_shape)\n",
    "    def call(self, x):\n",
    "        mean = tf.keras.backend.mean(x, axis=-1, keepdims=True)\n",
    "        std = tf.keras.backend.std(x, axis=-1, keepdims=True)\n",
    "        return self.gamma * (x - mean) / (std + self.eps) + self.beta\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return input_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, n_heads, ddf, dropout_rate=0.1):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "        \n",
    "        self.mha = MutilHeadAttention(d_model, n_heads)\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, ddf)\n",
    "        \n",
    "        self.layernorm1 = LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = LayerNormalization(epsilon=1e-6)\n",
    "        \n",
    "        self.dropout1 = tf.keras.layers.Dropout(dropout_rate)\n",
    "        self.dropout2 = tf.keras.layers.Dropout(dropout_rate)\n",
    "        \n",
    "    def call(self, inputs, training, mask):\n",
    "        # 多头注意力网络\n",
    "        att_output, _ = self.mha(inputs, inputs, inputs, mask)\n",
    "        att_output = self.dropout1(att_output, training=training)\n",
    "        out1 = self.layernorm1(inputs + att_output)  # (batch_size, input_seq_len, d_model)\n",
    "        # 前向网络\n",
    "        ffn_output = self.ffn(out1)\n",
    "        ffn_output = self.dropout2(ffn_output, training=training)\n",
    "        out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
    "        return out2\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "encoder层测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 43, 512])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
    "sample_encoder_layer_output = sample_encoder_layer(\n",
    "tf.random.uniform((64, 43, 512)), False, None)\n",
    "\n",
    "sample_encoder_layer_output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解码层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每个编码层包含以下子层：\n",
    "- Masked muti-head attention（带padding掩码和look-ahead掩码）\n",
    "- Muti-head attention（带padding掩码）value和key来自encoder输出，query来自Masked muti-head attention层输出\n",
    "- Point wise feed forward network\n",
    "\n",
    "每个子层中都有残差连接，并最后通过一个正则化层。残差连接有助于避免深度网络中的梯度消失问题。\n",
    "\n",
    "每个子层输出是LayerNorm(x + Sublayer(x))，规范化是在d_model维的向量上。Transformer一共有n个解码层。\n",
    "\n",
    "当Q从解码器的第一个注意块接收输出，并且K接收编码器输出时，注意权重表示基于编码器输出给予解码器输入的重要性。 换句话说，解码器通过查看编码器输出并自我关注其自己的输出来预测下一个字。\n",
    "\n",
    "ps：因为padding在后面所以look-ahead掩码同时掩padding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads, dff, drop_rate=0.1):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "        \n",
    "        self.mha1 = MutilHeadAttention(d_model, num_heads)\n",
    "        self.mha2 = MutilHeadAttention(d_model, num_heads)\n",
    "        \n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "        \n",
    "        self.layernorm1 = LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm3 = LayerNormalization(epsilon=1e-6)\n",
    "        \n",
    "        self.dropout1 = layers.Dropout(drop_rate)\n",
    "        self.dropout2 = layers.Dropout(drop_rate)\n",
    "        self.dropout3 = layers.Dropout(drop_rate)\n",
    "        \n",
    "    def call(self,inputs, encode_out, training, \n",
    "             look_ahead_mask, padding_mask):\n",
    "        # masked muti-head attention\n",
    "        att1, att_weight1 = self.mha1(inputs, inputs, inputs,look_ahead_mask)\n",
    "        att1 = self.dropout1(att1, training=training)\n",
    "        out1 = self.layernorm1(inputs + att1)\n",
    "        # muti-head attention\n",
    "        att2, att_weight2 = self.mha2(encode_out, encode_out, inputs, padding_mask)\n",
    "        att2 = self.dropout2(att2, training=training)\n",
    "        out2 = self.layernorm2(out1 + att2)\n",
    "        \n",
    "        ffn_out = self.ffn(out2)\n",
    "        ffn_out = self.dropout3(ffn_out, training=training)\n",
    "        out3 = self.layernorm3(out2 + ffn_out)\n",
    "        \n",
    "        return out3, att_weight1, att_weight2\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试解码层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder_layer = DecoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_decoder_layer_output, _, _ = sample_decoder_layer(\n",
    "tf.random.uniform((64, 50, 512)), sample_encoder_layer_output,\n",
    "    False, None, None)\n",
    "sample_decoder_layer_output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 编码器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "编码器包含：\n",
    "- Input Embedding\n",
    "- Positional Embedding\n",
    "- N个编码层\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(layers.Layer):\n",
    "    def __init__(self, n_layers, d_model, n_heads, ddf,\n",
    "                input_vocab_size, max_seq_len, drop_rate=0.1):\n",
    "        super(Encoder, self).__init__()\n",
    "        \n",
    "        self.n_layers = n_layers\n",
    "        self.d_model = d_model\n",
    "        \n",
    "        self.embedding = layers.Embedding(input_vocab_size, d_model)\n",
    "        self.pos_embedding = positional_encoding(max_seq_len, d_model)\n",
    "        \n",
    "        self.encode_layer = [EncoderLayer(d_model, n_heads, ddf, drop_rate)\n",
    "                            for _ in range(n_layers)]\n",
    "        \n",
    "        self.dropout = layers.Dropout(drop_rate)\n",
    "    def call(self, inputs, training, mark):\n",
    "        \n",
    "        seq_len = inputs.shape[1]\n",
    "        word_emb = self.embedding(inputs)\n",
    "        word_emb *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        emb = word_emb + self.pos_embedding[:,:seq_len,:]\n",
    "        x = self.dropout(emb, training=training)\n",
    "        for i in range(self.n_layers):\n",
    "            x = self.encode_layer[i](x, training, mark)\n",
    "        \n",
    "        return x\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "编码器测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 120, 512])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_encoder = Encoder(2, 512, 8, 1024, 5000, 200)\n",
    "sample_encoder_output = sample_encoder(tf.random.uniform((64, 120)),\n",
    "                                      False, None)\n",
    "sample_encoder_output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解码器\n",
    "解码器包含以下部分：1、输出嵌入；2、位置编码；3、n个解码层\n",
    "\n",
    "输出嵌入和位置编码叠加后输入解码器，解码器最后的输出送给一个全连接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pdb\n",
    "# pdb.set_trace()\n",
    "class Decoder(layers.Layer):\n",
    "    def __init__(self, n_layers, d_model, n_heads, ddf,\n",
    "                target_vocab_size, max_seq_len, drop_rate=0.1):\n",
    "        super(Decoder, self).__init__()\n",
    "        \n",
    "        self.d_model = d_model\n",
    "        self.n_layers = n_layers\n",
    "        \n",
    "        self.embedding = layers.Embedding(target_vocab_size, d_model)\n",
    "        self.pos_embedding = positional_encoding(max_seq_len, d_model)\n",
    "        \n",
    "        self.decoder_layers= [DecoderLayer(d_model, n_heads, ddf, drop_rate)\n",
    "                             for _ in range(n_layers)]\n",
    "        \n",
    "        self.dropout = layers.Dropout(drop_rate)\n",
    "        \n",
    "    def call(self, inputs, encoder_out,training,\n",
    "             look_ahead_mark, padding_mark):\n",
    "    \n",
    "        seq_len = tf.shape(inputs)[1]\n",
    "        attention_weights = {}\n",
    "        h = self.embedding(inputs)\n",
    "        h *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        h += self.pos_embedding[:,:seq_len,:]\n",
    "        \n",
    "        h = self.dropout(h, training=training)\n",
    "#         print('--------------------\\n',h, h.shape)\n",
    "        # 叠加解码层\n",
    "        for i in range(self.n_layers):\n",
    "            h, att_w1, att_w2 = self.decoder_layers[i](h, encoder_out,\n",
    "                                                   training, look_ahead_mark,\n",
    "                                                   padding_mark)\n",
    "            attention_weights['decoder_layer{}_att_w1'.format(i+1)] = att_w1\n",
    "            attention_weights['decoder_layer{}_att_w2'.format(i+1)] = att_w2\n",
    "        \n",
    "        return h, attention_weights\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码器测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([64, 100, 512]), TensorShape([64, 8, 100, 100]))"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "sample_decoder = Decoder(2, 512,8,1024,5000, 200)\n",
    "sample_decoder_output, attn = sample_decoder(tf.random.uniform((64, 100)),\n",
    "                                            sample_encoder_output, False,\n",
    "                                            None, None)\n",
    "sample_decoder_output.shape, attn['decoder_layer1_att_w1'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建Transformer\n",
    "Transformer包含编码器、解码器和最后的线性层，解码层的输出经过线性层后得到Transformer的输出\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Transformer(tf.keras.Model):\n",
    "    def __init__(self, n_layers, d_model, n_heads, diff,\n",
    "                input_vocab_size, target_vocab_size,\n",
    "                max_seq_len, drop_rate=0.1):\n",
    "        super(Transformer, self).__init__()\n",
    "        \n",
    "        self.encoder = Encoder(n_layers, d_model, n_heads,diff,\n",
    "                              input_vocab_size, max_seq_len, drop_rate)\n",
    "        \n",
    "        self.decoder = Decoder(n_layers, d_model, n_heads, diff,\n",
    "                              target_vocab_size, max_seq_len, drop_rate)\n",
    "        \n",
    "        self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
    "    def call(self, inputs, targets, training, encode_padding_mask, \n",
    "            look_ahead_mask, decode_padding_mask):\n",
    "        \n",
    "        encode_out = self.encoder(inputs, training, encode_padding_mask)\n",
    "        print(encode_out.shape)\n",
    "        decode_out, att_weights = self.decoder(targets, encode_out, training, \n",
    "                                               look_ahead_mask, decode_padding_mask)\n",
    "        print(decode_out.shape)\n",
    "        final_out = self.final_layer(decode_out)\n",
    "        \n",
    "        return final_out, att_weights\n",
    "        \n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformer测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 62, 512)\n",
      "(64, 26, 512)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 26, 8000])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_transformer = Transformer(\n",
    "n_layers=2, d_model=512, n_heads=8, diff=1024,\n",
    "input_vocab_size=8500, target_vocab_size=8000, max_seq_len=120\n",
    ")\n",
    "temp_input = tf.random.uniform((64, 62))\n",
    "temp_target = tf.random.uniform((64, 26))\n",
    "fn_out, _ = sample_transformer(temp_input, temp_target, training=False,\n",
    "                              encode_padding_mask=None,\n",
    "                               look_ahead_mask=None,\n",
    "                               decode_padding_mask=None,\n",
    "                              )\n",
    "fn_out.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.实验设置\n",
    "### 设置超参\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "\n",
    "input_vocab_size = tokenizer_pt.vocab_size + 2\n",
    "target_vocab_size = tokenizer_en.vocab_size + 2\n",
    "max_seq_len = 40\n",
    "dropout_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化器\n",
    "带自定义学习率调整的Adam优化器\n",
    "$$\\Large{lrate = d_{model}^{-0.5} * min(step{\\_}num^{-0.5}, step{\\_}num * warmup{\\_}steps^{-1.5})}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "        \n",
    "        self.d_model = tf.cast(d_model, tf.float32)\n",
    "        self.warmup_steps = warmup_steps\n",
    "    \n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps ** -1.5)\n",
    "        \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "learing_rate = CustomSchedule(d_model)\n",
    "optimizer = tf.keras.optimizers.Adam(learing_rate, beta_1=0.9, \n",
    "                                    beta_2=0.98, epsilon=1e-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'train step')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XXW57/HPk6RJ2qZJ2yRt0zGdS8pMGARkHgoiVQEper14hMNVQTkOKOg5Dni5R/QoiIJeFAQ9YikgWrjMgsyUpgVKB1uS3dKmdNhJx6Rj0uf+sVba3ZBhN8nK3km+79crr6z9W2v91rN30jxd6/dbzzJ3R0REpKtlpDoAERHpnZRgREQkEkowIiISCSUYERGJhBKMiIhEQglGREQioQQjIiKRUIIREZFIKMGIiEgkslIdQCoVFRV5aWlpqsMQEelRFixYUOPuxe1t16cTTGlpKRUVFakOQ0SkRzGz95PZTpfIREQkEkowIiISCSUYERGJhBKMiIhEItIEY2YzzGy5mVWa2Y0trM8xswfD9fPMrDRh3U1h+3IzOz+h/V4z22hmi1s55jfMzM2sKIr3JCIiyYkswZhZJnAncAFQBlxhZmXNNrsK2Ozuk4DbgFvDfcuAWcB0YAZwV9gfwH1hW0vHHAOcB6zu0jcjIiKHLMozmBOASnePufseYDYws9k2M4H7w+WHgbPNzML22e6+291XApVhf7j7S8CmVo55G/AtQI/pFBFJsSgTzChgTcLr6rCtxW3cvQHYChQmue9BzGwmsNbd3+lc2OnL3Zkzfw11uxtSHYqISLt6xSC/mQ0AvgN8L4ltrzGzCjOriMfj0QfXhd5es4VvPbKIbz+8KNWhiIi0K8oEsxYYk/B6dNjW4jZmlgUUALVJ7ptoIjAeeMfMVoXbLzSzEc03dPe73b3c3cuLi9utdJBWVm/aAcCzSzekOBIRkfZFmWDmA5PNbLyZZRMM2s9tts1c4Mpw+VLgeXf3sH1WOMtsPDAZeLO1A7n7u+4+zN1L3b2U4JLase6+vmvfUmpVxesB2NO4j9W1O1IcjYhI2yJLMOGYynXA08AyYI67LzGzm83s4nCze4BCM6sEvg7cGO67BJgDLAWeAq5190YAM/sz8Dow1cyqzeyqqN5DuqmK12EWLD+1ZF1qgxERaYcFJwx9U3l5ufekYpcX/OJlhufnEN++m+ysDB798impDklE+iAzW+Du5e1t1ysG+fuCffuclTV1TCzO48IjSnhr9RbWbd2Z6rBERFqlBNNDfLB1J7v27mNC8UBmHB7MXXhqca8aYhKRXkYJpoeIhQP8E4vzmFicx9Thg3jsnQ9SHJWISOuUYHqIqngdABOKBwIw85iRLFy9hfdr61MZlohIq5RgeohYvJ5BuVkU5+UA8Imjg8IGf31LZzEikp6UYHqIqngdE4rzsHCe8sjB/TlpwlAefauavjwTUETSlxJMDxGL1zOxaOBBbZ86ZjSranfw1potKYpKRKR1SjA9QN3uBtZv28XEYXkHtV9wxAhysjJ4dGFbVXRERFJDCaYHWBnOIJvQ7AxmUG4/zi0bzmOLPmB3Q2MqQhMRaZUSTA8QqwlmkDU/gwG4rHwMW3bs5ZklKoApIulFCaYHqNpYR4bBuMIBH1r30UlFjB7Snwfm6SGeIpJelGB6gKqaekYPGUBOVuaH1mVkGFecMJbXY7XEwntlRETSgRJMD1C1sY6JxQNbXX9Z+WiyMozZ89e0uo2ISHdTgklz+/Y5q2rrmVD84fGXJsMG5XLOYcN5eEG1BvtFJG0owaS5piKXE9tIMACfOXEsm+r3qACmiKQNJZg01/QUywltXCIDOHVSEeOLBnLvq6t0Z7+IpAUlmDTXNHDf3hlMRobxL6eU8s6aLSxcvbk7QhMRaZMSTJqritcxKDeLorzsdre95NjR5Odmcc8rK7shMhGRtinBpLlYvP6gIpdtGZiTxRUnjuWpxetZs2lHN0QnItI6JZg0F4vXtzlFubnPn1xKhhn3vbYquqBERJIQaYIxsxlmttzMKs3sxhbW55jZg+H6eWZWmrDuprB9uZmdn9B+r5ltNLPFzfr6qZn908wWmdmjZjY4yvfWHfYXuWxn/CVRSUF/LjyihAfnr2HLjj0RRici0rbIEoyZZQJ3AhcAZcAVZlbWbLOrgM3uPgm4Dbg13LcMmAVMB2YAd4X9AdwXtjX3LHC4ux8JrABu6tI3lAIr9z8mOfkzGIAvnTGRut0NOosRkZSK8gzmBKDS3WPuvgeYDcxsts1M4P5w+WHgbAsGG2YCs919t7uvBCrD/nD3l4BNzQ/m7s+4e0P48g1gdFe/oe524DHJyZ/BABxWks85hw3n3ldWsn3X3ihCExFpV5QJZhSQWLukOmxrcZswOWwFCpPcty1fAJ5saYWZXWNmFWZWEY/HD6HL7heLt17ksj1fPXsS23Y18Mc33o8gMhGR9vW6QX4z+y7QAPyppfXufre7l7t7eXFxcfcGd4iq4vWMGdpykcv2HDl6MKdPKeZ3L69kx56G9ncQEeliUSaYtcCYhNejw7YWtzGzLKAAqE1y3w8xs88DFwGf9V5wO3tVvO5DDxk7FF85axKb6vfw3zqLEZEUiDLBzAcmm9l4M8smGLSf22ybucCV4fKlwPNhYpgLzApnmY0HJgNvtnUwM5sBfAu42N17/E0g+/Y5K2vqD2kGWXPlpUP56OQifv2PKrZpLEZEullkCSYcU7kOeBpYBsxx9yVmdrOZXRxudg9QaGaVwNeBG8N9lwBzgKXAU8C17t4IYGZ/Bl4HpppZtZldFfb1K2AQ8KyZvW1mv4nqvXWHtVt2srth3yEP8Df37RnT2LxjL799KdZFkYmIJCcrys7d/QngiWZt30tY3gVc1sq+twC3tNB+RSvbT+pUsGkmVtOxKcrNHT6qgIuOLOF3L6/kcx8Zx7BBuV0RnohIu3rdIH9vUbWxY1OUW/LN86ayt3Efv/x7Zaf7EhFJlhJMmorVJF/ksj2lRQO5/Pgx/PnN1awMz4xERKKmBJOmghpkyRW5TMb150wmJyuDW/7f0i7pT0SkPUowaaoqXtfuQ8YOxbBBuXz17Mk8t2wjLyzf2GX9ioi0RgkmDdXtbmDDtt2dmqLckn85ZTwTigbyo8eWsqdhX5f2LSLSnBJMGjrwFMuuO4MByM7K4D8+Xkaspp77XtNDyUQkWkowaSgWVlHuihlkzZ05dRhnTRvGL557j3Vbd3Z5/yIiTZRg0lBVJ4pcJuP7Hy+j0Z3/+OsSekFFHRFJU0owaSjWiSKXyRhXOJCvnTOF55Zt4MnF6yM5hoiIEkwaqorXdfkAf3NXnTqe6SPz+d7flrB1h+qUiUjXU4JJM01FLjtTRTkZWZkZ3HrJkWzesYf/88SySI8lIn2TEkyaaSpyOXFYtGcwENQpu/qj43mwYg0v/FP3xohI11KCSTP7H5Mc8RlMk6+dM4VpIwZxw8OLqK3b3S3HFJG+QQkmzUQ5Rbkluf0yue3yo9m2cy83/eVdzSoTkS6jBJNmYjV15HdRkctkHVaSzw3nT+WZpRt4qKK6244rIr2bEkyaqdpYz4QuLHKZrKtOHc9HJhTyw8eW7L9MJyLSGUowaSZWE/0U5ZZkZBg/v/wocvpl8uX/XsjOPY3dHoOI9C5KMGlk+669bNi2u0urKB+KkoL+3H750azYuJ1//+tijceISKcowaSRlV30mOTOOG1KMV85azKPLKxmTsWalMUhIj1fpAnGzGaY2XIzqzSzG1tYn2NmD4br55lZacK6m8L25WZ2fkL7vWa20cwWN+trqJk9a2bvhd+HRPneolC1v4py918iS3T92ZM5dVIR//G3JSxeuzWlsYhIzxVZgjGzTOBO4AKgDLjCzMqabXYVsNndJwG3AbeG+5YBs4DpwAzgrrA/gPvCtuZuBP7u7pOBv4eve5RYvJ4Mg7ERFblMVmaGcfusoykamM2//qGCjdt2pTQeEemZojyDOQGodPeYu+8BZgMzm20zE7g/XH4YONuC6VMzgdnuvtvdVwKVYX+4+0vAphaOl9jX/cAnuvLNdIdYvJ6xERa5PBRFeTn89spytuzYyzV/XMCuvRr0F5FDE2WCGQUkXsSvDtta3MbdG4CtQGGS+zY33N3XhcvrgeEtbWRm15hZhZlVxOPxZN5Htwkek5zay2OJpo8s4LbLj+LtNVu48ZFFGvQXkUPSKwf5PfhL2OJfQ3e/293L3b28uLi4myNrXWNY5DKVA/wtmXF4Cd88bwp/ffsDfvl8ZarDEZEeJMoEsxYYk/B6dNjW4jZmlgUUALVJ7tvcBjMrCfsqAXpU9cYPwiKX6XQG0+TaMyfxqWNH8fNnVzD7zdWpDkdEeogoE8x8YLKZjTezbIJB+7nNtpkLXBkuXwo8H559zAVmhbPMxgOTgTfbOV5iX1cCf+uC99BturvI5aEwM2695EhOn1LMdx59l2eXbkh1SCLSA0SWYMIxleuAp4FlwBx3X2JmN5vZxeFm9wCFZlYJfJ1w5pe7LwHmAEuBp4Br3b0RwMz+DLwOTDWzajO7Kuzrx8C5ZvYecE74usdoKnLZHWX6O6JfZgZ3ffZYjhhVwHUPLGT+qpbmWYiIHGB9eeC2vLzcKyoqUh0GAN999F0ee+cD3vn+ed1eh+xQ1Nbt5rLfvE5N3W4e+NeTOHxUQapDEpFuZmYL3L28ve165SB/TxSL1zNxWPcXuTxUhXk53P+FE8jLyeJ/3DOPZeu2pTokEUlTSjBpoipex4Si9Lw81tyYoQP48zUnkZuVyWd/N4/l67enOiQRSUNKMGlg+669bNyeuiKXHTGucCB/vuYksjKMz/7uDSo3KsmIyMGUYNLA/gH+NJyi3JbxRUGSAWPW3W+w9ANdLhORA5Rg0kCspqnIZc85g2kysTiPB//XSWRnZnD53a9TodllIhJSgkkDsXg9mRmW8iKXHTWxOI+HvnQyRXk5/I975vHiivQqwSMiqaEEkwaq4nWMGdI/LYpcdtSowf2Z878+wviiPK6+fz6PL/og1SGJSIopwaSBWLy+x42/tKR4UA6zrzmJo0YP5roH3uI3L1apQKZIH6YEk2KN+5xYTX2PmkHWloL+/fjvq0/koiNL+PGT/+Q7j77L3sZ9qQ5LRFIgK9UB9HUfbNnJnjQtctlRuf0yuWPWMYwrHMCdL1RRvXknd372WPJz+6U6NBHpRjqDSbF0eUxyV8vIMG44fxo/ueRIXq+q5ZK7XiMWvlcR6RuUYFKsKrwHprdcImvu08eP4Q9fOIGaut3M/NWrqsQs0ocowaRYLF5HQf9+FA7MTnUokTl5UhGPfeVUSosG8q9/qODnz65g3z4N/ov0dkowKRY8Jnlg2he57KzRQwbw0Bc/wqXHjeaOv7/HVffPZ3P9nlSHJSIRSirBmNmxZvZVM/uKmR0bdVB9SSxe32OKXHZWbr9MfnrpkfzoE4fzamUtF/ziZd6I1aY6LBGJSLsJxsy+B9wPFAJFwO/N7N+jDqwvaCpyOXFY7xx/aYmZ8bmTxvGXL59M/+xMrvjtG/z8meU0aCqzSK+TzBnMZ4Hj3f377v594CTgc9GG1Tc0FbnsK2cwiQ4fVcDjXzmVS44dzR3PV3L53W9QvXlHqsMSkS6UTIL5AMhNeJ0DrI0mnL6lqcjlpD50BpNoYE4W/3XZUfxi1tEsX7+dGbe/zIPzV+vuf5FeIpkEsxVYYmb3mdnvgcXAFjO7w8zuiDa83q1qY1jkcmjfTDBNZh49iiev/yiHj8rn24+8y+d/P591W3emOiwR6aRkEsyjwHeAF4B/AN8F/gYsCL9aZWYzzGy5mVWa2Y0trM8xswfD9fPMrDRh3U1h+3IzO7+9Ps3sbDNbaGZvm9krZjYpifeWUrGaOsYOHUB2libzjRk6gAeuPokfXjydN1du4rzbXmJOxRqdzYj0YO2WinH3+82sPzDW3Zcn27GZZQJ3AucC1cB8M5vr7ksTNrsK2Ozuk8xsFnArcLmZlQGzgOnASOA5M5sS7tNan78GZrr7MjP7MvDvwOeTjTcVqjbWM6Gob5+9JMrIMK48uZQzphZzw0OL+NbDi3jsnQ/40czDKdXnJNLjJDOL7OPA28BT4eujzWxuEn2fAFS6e8zd9wCzgZnNtplJMEMN4GHgbAtuCJkJzHb33e6+EqgM+2urTwfyw+UCgrGjtNW4z1lZ23uKXHalcYUDmX3NSdw8czpvr97Cebe/xC+ee4/dDY2pDk1EDkEy12Z+QPCHfQuAu78NTEhiv1HAmoTX1WFbi9u4ewPBeE9hG/u21efVwBNmVk0wy+3HScSYMk1FLntbDbKukpFh/M+PlPLcN07nvLLh3PbcCi64/WVeq6xJdWgikqRkEsxed9/arC0db1r4GnChu48Gfg/8vKWNzOwaM6sws4p4PHVPXqwMCz/2pirKURien8uvPnMs93/hBBrd+czv5nHdAws1pVmkB0gmwSwxs88AmWY22cx+CbyWxH5rgTEJr0fz4enN+7cxsyyCS1u1bezbYruZFQNHufu8sP1B4OSWgnL3u9293N3Li4uLk3gb0Wi6B2aiLpEl5fQpxTz9b6dx/dmTeW7ZBs762Yv89Ol/Ure7IdWhiUgrkkkwXyEYbN8NPEBwGev6JPabD0w2s/Fmlk0waN987GYucGW4fCnwvAfThuYCs8JZZuOBycCbbfS5GShImAhwLrAsiRhTpioscjm0Fxe57Gq5/TL52rlTeP4bZ3Dh4SO484UqzvyvfzBn/hoaVTxTJO0kk2A+5u7fdffjw69/By5ub6dwTOU64GmCP/Zz3H2Jmd1sZk373wMUmlkl8HXgxnDfJcAcYCnB5IJr3b2xtT7D9n8FHjGzdwjGYG5I9kNIhVgfKXIZhZGD+3P7rGN49MsnM2ZIf771yCIu+uUrPP/PDZrWLJJGrL1/kGa20N2Pba+tJyovL/eKioqUHPv4W57j9CnF/NdlR6Xk+L2Fu/PYonX87JnlvF+7g+PGDeGG86dy0oTCVIcm0muZ2QJ3L29vu1bvgzGzC4ALgVHN7tjPB3ThuxO279pLfPtuTVHuAmbGxUeN5ILDRzCnYg13/P09Zt39Bh+dXMQN50/lyNGDUx2iSJ/V1iWyD4AKYBcH7tpfQDDmcX4b+0k7DgzwawZZV+mXmcFnTxzHizecyXcunMa7a7dy8a9e5er75/PW6s2pDk+kT2r1DMbd3wHeMbMH3H0vgJkNAca4u/7FdkJVOEVZM8i6Xm6/TK45bSJXnDCWe19Zxb2vruSTd73GqZOKuO6sSZw4fqjGvUS6STKD/M+aWb6ZDQUWAr81s9sijqtXi8VV5DJqg3L7cf05k3n1xrO48YJp/HP9Nmbd/QaX/eZ1Xli+UZMBRLpBMgmmwN23AZ8C/uDuJwJnRxtW71YVV5HL7pKXk8UXT5/IK98+ix9ePJ21W3byL7+fz4V3vMLDC6pVfkYkQsn8hcsysxLg08DjEcfTJwSPSdbZS3fK7ZfJlSeX8uINZ/KTS46kcd8+vvnQO5zy4xe44+/vUVu3O9UhivQ6ySSYmwnuO6l09/lmNgF4L9qweq+mIpcTh2mAPxWyszL49PFjePrfTuMPXziB6SPz+fmzKzj5x89z4yOLWLFhe6pDFOk1kinX/xDwUMLrGHBJlEH1Zms3B0UudQaTWmbGaVOKOW1KMZUbt3PPK6v4y8JqZs9fw0cmFPLZk8ZyXtkIXcYU6YR2E4x0rarwMck6g0kfk4YN4j8/dQQ3nD+VP7+5mgfmrea6B96iKC+bT5eP4YoTxjJm6IBUhynS4yjBdLOqjWEVZZ3BpJ2hA7O59sxJfPH0ibz0Xpw/vbGa37xYxa9frOK0ycV89sSxnDVtGFmZOqsRSYYSTDeL1dQzeICKXKazzAzjzKnDOHPqMD7YspMH569h9vzVXPPHBRTlZfOJo0dxyXGjOawkv/3ORPqwdhOMmeUQjLmUJm7v7jdHF1bvVbWxjglFKnLZU4wc3J+vnTuFr5w1iReWx3l4wRruf30Vv3tlJWUl+Vxy3GhmHj2SorycVIcqknaSOYP5G0GJ/gUEJfulE2I19Zw+JXXPoZGOycrM4Nyy4ZxbNpxN9Xt47J0PeGRhNT96fCn/+cQyzphazCePGc1Z04bRPzsz1eGKpIVkEsxod58ReSR9wLawyKVqkPVsQwdmc+XJpVx5cikrNmznkYXV/PWttTy3bCMDsjM557DhXHRkCadPLSYnS8lG+q5kEsxrZnaEu78beTS9XFORS1VR7j2mDB/ETRccxrfOn8a8WC2PLVrHk4vXMfedDxiUm8V5ZSO46KgSTp1URD9NDpA+JpkEcyrweTNbSXCJzAB39yMjjawXiu0vcqkzmN4mM8M4eVIRJ08q4uaZ03m1sobHF63j6SXreWRhNYMH9OO8suGcVzaCUycXkdtPZzbS+yWTYC6IPIo+oipeFxa51D0VvVm/zAzOmDqMM6YO45ZPHs5LK2p4fNEHPPnueuZUVDMgO5PTpxRz3vThnDV1OAUD+qU6ZJFItPXAsfywyKVqZ3SRWLxeRS77mJyszP2TA/Y07OONWC3PLF3PM0s28OTi9WRlGCdOGMp5ZSM4p2w4owb3T3XIIl2m1Ucmm9nj7n5ReGnMCS6NNXF3n9AdAUapux+ZfN5tLzJ26AB+d+Xx3XZMSU/79jnvVG/hmaUbeGbJeqrC8bkpw/M4c+owTp9aTPm4ofrPiKSlTj8y2d0vCr+P70QQM4BfAJnA79z9x83W5wB/AI4DaoHL3X1VuO4m4CqgEfiquz/dVp8W3Fjyv4HLwn1+7e6Jj3pOqcZ9zqraHZwxdViqQ5E0kJFhHDN2CMeMHcK3Z0yjcmMd/1i+kReWb+TeV1fyf1+KkZeTxamTijhjajFnTB3GiILcVIctckiSupM/fJLlZGD/b7i7v9TOPpnAncC5QDUw38zmuvvShM2uAja7+yQzmwXcClxuZmXALGA6MBJ4zsymhPu01ufngTHANHffZ2Zp9Ze8qcilnmIpLZk0LI9Jw/K4+qMTqNvdwGuVNbywPM6Lyzfy1JL1ABxWks9pk4s4ZVIRx5cO1f02kvaSuZP/auB6YDTwNnAS8DpwVju7nkBQ4j8W9jMbmAkkJpiZwA/C5YeBX4VnIjOB2e6+G1hpZpVhf7TR55eAz7j7PgB339jee+tOTY9JnqAZZNKOvJwszps+gvOmj8DdWbGhjheWb+QfCWc32ZkZHDtuMKdMLOKUyUUcOapANdIk7SRzBnM9cDzwhrufaWbTgP+TxH6jgDUJr6uBE1vbxt0bzGwrUBi2v9Fs31Hhcmt9TiQ4+/kkECe4rJY2z62p0hRl6QAzY+qIQUwdMYgvnj6RHXsamL9qM69V1vBKZQ0/f24FP3t2BYNysjhxQiGnTCrklElFTCrOIyND5YgktZJJMLvcfZeZYWY57v5PM5saeWSHLocg1nIz+xRwL/DR5huZ2TXANQBjx47ttuCq4ipyKZ03IDuL06cU7y83tKl+D69X1fJqVQ2vVtbw3LINAAwZ0I/jS4dywvihnDi+kMNKBukMR7pdMgmm2swGA38FnjWzzcD7Sey3lmBMpMnosK2lbarNLAsoIBjsb2vf1tqrgb+Ey48Cv28pKHe/G7gbgllkSbyPLhGL16lEv3S5oQOz+diRJXzsyBIA1mzaweuxWuav3MSbqzbxzNIg4QzMzuS40qGcOD5IOkeOLlAZG4lcMk+0/GS4+AMze4EgCTyVRN/zgclmNp4gCcwCPtNsm7nAlQRjOpcCz7u7m9lc4AEz+znBIP9k4E2CqdKt9flX4ExgJXA6sCKJGLtNrKaeM1TkUiI2ZugAxgwdwKfLg/+Hbdi2izdXbtr/9dOnlwPBo6OPHjOY8nFDwtlsg1URWrpcmwkmnAm2xN2nAbj7i8l2HI6pXAc8TTCl+F53X2JmNwMV7j4XuAf4YziIv4kgYRBuN4dg8L4BuNbdG8OYPtRneMgfA38ys68BdcDVycYataYilxrgl+42PD+Xjx81ko8fNRKAzfV7mL8qSDbzV23i7pdiNOwLTuTHDh3AMWMHc2yYcA4ryVf9NOmUVm+03L+B2d+Ar7j76u4Jqft0142Wb6/ZwifufJW7P3cc500fEfnxRJK1a28ji9duZeHqzby1egsLV29mw7bgqRw5WRkcObogOMMZM5ijxgympCBXzzKSzt9omWAIsMTM3gTqmxrd/eJOxNen7H9Mss5gJM3k9sukvHQo5aVD97d9sGUnb63ewlurN7Nw9Wbue3UVdzfuA6AoL5vDRxVwRNPX6AJG5CvpSMuSSTD/EXkUvVysRkUupecYObg/Iwf33z9xYHdDI8vWbefd6i0sqt7Ku2u38vJ7NTSGl9aK8nI4YlQ+R4wezBGjCjhydAHD81V1QJJLMBe6+7cTG8zsViDp8Zi+rmpjPeNU5FJ6qJysTI4eM5ijxwze37ZzTyPL1m/j3eqtLKreyuK1W3lxxXuEOYfiQTkcVpLPYSWDKCvJ57CSfCYUDdRU6T4mmQRzLvDtZm0XtNAmrYjV1OkhY9Kr9M/O5NixQzh27JD9bTv2NLBs3bYw4Wxj2bpt3FtVw97GIOtkZ2UwZXgeh43ID5NPPmUl+XpcQS/WVrn+LwFfBiaY2aKEVYOAV6MOrLdo3OesqtnBmSpyKb3cgOwsjhs3lOPGHRjP2dOwj6p4HcvWbQu/tvP8Pzfy0ILq/duMLMjlsJJ8ppUMYsrwQUweNoiJwwbqPp1eoK0zmAeAJ4H/BG5MaN/u7psijaoXqd68gz2N+3QGI31SdlbG/rOVJu5OfPtuloYJpyn5/GNFfP+4ToZBaeFAJg/PC5LO8EFMGZ7HhKI8XWruQdoq178V2Apc0X3h9D6x8DkfqkEmEjAzhuXnMiw/96DHV+xuaGRlTT0rNtTx3obtrNiwnfc21PHs0g37x3YyM4zSwgEfSjrjiwaqunQaSqpcv3ScqiiLJCcnK5NpI/KZNiL/oPZdexuJxet5b2OQdFZsCC65PbVkPYm38Y0a3J8JxQOZUDSQCcV5TCzOY0LxQEbk56rwZ4oowURMRS5FOie3XyZlI/MpG9kEY95wAAATPUlEQVRy4onV1AXf43VUxet5eEE19Xsa92/Xv18m44sGBsmnOI+JxQODs57igeTl6E9glPTpRiwWr9PlMZEItJZ43J2N23dTFW9KPEESWlS9lSfeXbf/chsE9/CUFg5gbOEAxg0dSGnRAMYOHUBp4UAGD+inG0g7SQkmYlXxes6cqiKXIt3FzBien8vw/FxOnlh00LpdextZvWnH/rOd1bU7WFVbz+tVtfxl4cHF3gflZjGucADjCgcyLkw6YwsHMK5wAMMH6bJbMpRgIrR1515q6nYzcZjOYETSQW6/TKYMD6ZDN7drbyNrNu3g/TDprN60g1W1O1iyditPL16/vygoBHXaxg4Nks3oIQMYPaR/+BUsF/TX2Q8owUQq1jTAr+fAiKS93H6ZTA5npzXX0LiPD7bs4v1N9ayq3cHq2uD7mk07eCO2ibrdDQdtPygni1EJCScx+YwZMoD8/ll9IgEpwUSoaYqyZpCJ9GxZmRmMDcdqPjr54HXuztade6nevJPqzTvC703LO3gjVttuAho1uD8lg3MpKejPyMG5DBuUS2YvuASnBBOhqngdWRnGuEIVuRTprcyMwQOyGTwgqDTdXEcSUGaGMXxQDiWD+1NSkBsUIC3IpWRwf0YWBMmocGB22p8FKcFEKBavZ+zQAXpok0gflkwC2rargXVbd7Juyy4+aPZ98dqtPLN0A3sa9h20X3ZWBiUFuUECKjhwBjQinOAwvCCHooE5KZ2MoAQToaDIpS6PiUjrzIyC/v0o6N/vQzeZNnF3NtXvYd3WXXywZWfwPUxA67buZN7KTazftmt/qZ0mWRnGsEE5DC/I3Z94RoTLJ08sZFjEj1VQgomIilyKSFcxMwrzcijMy2nxLAiCvzk1dbtZv3UX67ftYsO2XQctr9iwnZffq9l/Oe4PXzhBCaanaipyqZssRaQ7ZGYcuP/nqDa2q9vdwPqtuygpiP6hcEowETlQg0xTlEUkfeTlZDGpm+7Ni3T02cxmmNlyM6s0sxtbWJ9jZg+G6+eZWWnCupvC9uVmdv4h9HmHmdVF9Z6SpSnKItLXRZZgzCwTuJPg6ZdlwBVmVtZss6uAze4+CbgNuDXctwyYBUwHZgB3mVlme32aWTkwhDRQFa9niIpcikgfFuUZzAlApbvH3H0PMBuY2WybmcD94fLDwNkWTOyeCcx2993uvhKoDPtrtc8w+fwU+FaE7ylpVXHNIBORvi3KBDMKWJPwujpsa3Ebd28geMBZYRv7ttXndcBcd1/XVlBmdo2ZVZhZRTweP6Q3dChi8XomavxFRPqwXnEHoJmNBC4Dftnetu5+t7uXu3t5cXE0VY6bilzqDEZE+rIoE8xaYEzC69FhW4vbmFkWUADUtrFva+3HAJOASjNbBQwws8queiOHSkUuRUSiTTDzgclmNt7MsgkG7ec222YucGW4fCnwvLt72D4rnGU2HpgMvNlan+7+/9x9hLuXunspsCOcOJASVeEMMpXpF5G+LLL7YNy9wcyuA54GMoF73X2Jmd0MVLj7XOAe4I/h2cYmgoRBuN0cYCnQAFzr7o0ALfUZ1XvoqFhY5HLsUBW5FJG+K9IbLd39CeCJZm3fS1jeRTB20tK+twC3JNNnC9uk9NQhFq9nbKGKXIpI36a/gBGoitcxoUiXx0Skb1OC6WINjft4v3YHE4dpgF9E+jYlmC5WvXlnUORSZzAi0scpwXSxWI2KXIqIgBJMl2sqcqky/SLS1ynBdLGqeB1DBvRjiIpcikgfpwTTxari9Tp7ERFBCabLxeJ1Gn8REUEJpktt3bGXmro9KnIpIoISTJeqCmeQ6RKZiIgSTJc68JhkXSITEVGC6UIqcikicoASTBeqitepyKWISEh/CbtQTFOURUT2U4LpIg2N+1hVW6/xFxGRkBJMF6nevJO9ja4ilyIiISWYLtJU5FJl+kVEAkowXaRqYzhFWWcwIiKAEkyXidXUMXRgtopcioiEIk0wZjbDzJabWaWZ3djC+hwzezBcP8/MShPW3RS2Lzez89vr08z+FLYvNrN7zaxflO+tuaqN9Uwo0uUxEZEmkSUYM8sE7gQuAMqAK8ysrNlmVwGb3X0ScBtwa7hvGTALmA7MAO4ys8x2+vwTMA04AugPXB3Ve2tJrEZFLkVEEkV5BnMCUOnuMXffA8wGZjbbZiZwf7j8MHC2mVnYPtvdd7v7SqAy7K/VPt39CQ8BbwKjI3xvB2kqcql7YEREDogywYwC1iS8rg7bWtzG3RuArUBhG/u222d4aexzwFOdfgdJqtr/mGQlGBGRJr1xkP8u4CV3f7mllWZ2jZlVmFlFPB7vkgMeeEyyLpGJiDSJMsGsBcYkvB4dtrW4jZllAQVAbRv7ttmnmX0fKAa+3lpQ7n63u5e7e3lxcfEhvqWWVYVFLseoyKWIyH5RJpj5wGQzG29m2QSD9nObbTMXuDJcvhR4PhxDmQvMCmeZjQcmE4yrtNqnmV0NnA9c4e77InxfHxKL1zFORS5FRA6SFVXH7t5gZtcBTwOZwL3uvsTMbgYq3H0ucA/wRzOrBDYRJAzC7eYAS4EG4Fp3bwRoqc/wkL8B3gdeD+YJ8Bd3vzmq95eoKl6v8RcRkWYiSzAQzOwCnmjW9r2E5V3AZa3sewtwSzJ9hu2RvpfWNDTu4/3aes4+bFgqDi8ikrZ0TaeT9he51BmMiMhBlGA6qSoeFrnUDDIRkYMowXRS0xRlFbkUETmYEkwnVcVV5FJEpCVKMJ0Ui6vIpYhIS5RgOqkqXqcBfhGRFijBdMLWHXuprd+jKsoiIi1QgumEpiKXOoMREfkwJZhOqNrYVEVZZzAiIs0pwXRCrKaefpkqciki0hIlmE6o2ljH2KEqciki0hL9ZeyEWI2KXIqItEYJpoOailxqgF9EpGVKMB20JixyqQF+EZGWKcF0UCyuKcoiIm1RgukgVVEWEWmbEkwHxeL1FA7MZvAAFbkUEWmJEkwHVcXrNP4iItIGJZgOCqooa/xFRKQ1kSYYM5thZsvNrNLMbmxhfY6ZPRiun2dmpQnrbgrbl5vZ+e31aWbjwz4qwz4ju3a1Zcceauv3MHGYzmBERFoTWYIxs0zgTuACoAy4wszKmm12FbDZ3ScBtwG3hvuWAbOA6cAM4C4zy2ynz1uB28K+Nod9R6JKT7EUEWlXlGcwJwCV7h5z9z3AbGBms21mAveHyw8DZ5uZhe2z3X23u68EKsP+Wuwz3OessA/CPj8R1RvbP0V5mBKMiEhrokwwo4A1Ca+rw7YWt3H3BmArUNjGvq21FwJbwj5aO1aXqYqHRS6H9I/qECIiPV6fG+Q3s2vMrMLMKuLxeIf6KC0cwCePGUWWilyKiLQqyr+Qa4ExCa9Hh20tbmNmWUABUNvGvq211wKDwz5aOxYA7n63u5e7e3lxcXEH3hbMOmEsP7n0qA7tKyLSV0SZYOYDk8PZXdkEg/Zzm20zF7gyXL4UeN7dPWyfFc4yGw9MBt5src9wnxfCPgj7/FuE701ERNqR1f4mHePuDWZ2HfA0kAnc6+5LzOxmoMLd5wL3AH80s0pgE0HCINxuDrAUaACudfdGgJb6DA/5bWC2mf1v4K2wbxERSREL/vPfN5WXl3tFRUWqwxAR6VHMbIG7l7e3nUapRUQkEkowIiISCSUYERGJhBKMiIhEQglGREQi0adnkZlZHHi/g7sXATVdGE5XUVyHRnEdGsV1aNI1LuhcbOPcvd071ft0gukMM6tIZpped1Nch0ZxHRrFdWjSNS7onth0iUxERCKhBCMiIpFQgum4u1MdQCsU16FRXIdGcR2adI0LuiE2jcGIiEgkdAYjIiKRUILpADObYWbLzazSzG7shuOtMrN3zextM6sI24aa2bNm9l74fUjYbmZ2RxjbIjM7NqGfK8Pt3zOzK1s7Xjux3GtmG81scUJbl8ViZseF77Uy3Nc6EdcPzGxt+Lm9bWYXJqy7KTzGcjM7P6G9xZ9t+IiIeWH7g+HjItqLaYyZvWBmS81siZldnw6fVxtxpfTzCvfLNbM3zeydMLYfttWfBY/0eDBsn2dmpR2NuYNx3WdmKxM+s6PD9u783c80s7fM7PF0+KwO4u76OoQvgscEVAETgGzgHaAs4mOuAoqatf0EuDFcvhG4NVy+EHgSMOAkYF7YPhSIhd+HhMtDOhDLacCxwOIoYiF47s9J4T5PAhd0Iq4fAN9sYduy8OeWA4wPf56Zbf1sgTnArHD5N8CXkoipBDg2XB4ErAiPndLPq424Uvp5hdsakBcu9wPmhe+vxf6ALwO/CZdnAQ92NOYOxnUfcGkL23fn7/7XgQeAx9v67Lvrs0r80hnMoTsBqHT3mLvvAWYDM1MQx0zg/nD5fuATCe1/8MAbBE/6LAHOB551903uvhl4FphxqAd195cInt3T5bGE6/Ld/Q0PfvP/kNBXR+JqzUxgtrvvdveVQCXBz7XFn234P8mzgIdbeI9txbTO3ReGy9uBZcAoUvx5tRFXa7rl8wrjcXevC1/2C7+8jf4SP8uHgbPD4x9SzJ2IqzXd8rM0s9HAx4Dfha/b+uy75bNKpARz6EYBaxJeV9P2P86u4MAzZrbAzK4J24a7+7pweT0wvJ34ooy7q2IZFS53ZYzXhZco7rXwUlQH4ioEtrh7Q0fjCi9HHEPwP9+0+byaxQVp8HmFl3zeBjYS/AGuaqO//TGE67eGx+/yfwfN43L3ps/slvAzu83McprHleTxO/qzvB34FrAvfN3WZ99tn1UTJZie4VR3Pxa4ALjWzE5LXBn+jyctpgOmUyzAr4GJwNHAOuBnqQjCzPKAR4B/c/dtietS+Xm1EFdafF7u3ujuRwOjCf4XPS0VcTTXPC4zOxy4iSC+4wkue327u+Ixs4uAje6+oLuOeaiUYA7dWmBMwuvRYVtk3H1t+H0j8CjBP7oN4Wk14feN7cQXZdxdFcvacLlLYnT3DeEfhX3Abwk+t47EVUtwiSOrWXu7zKwfwR/xP7n7X8LmlH9eLcWVDp9XInffArwAfKSN/vbHEK4vCI8f2b+DhLhmhJcb3d13A7+n459ZR36WpwAXm9kqgstXZwG/II0+q8gGpnvrF5BFMDA3ngMDX9MjPN5AYFDC8msEYyc/5eCB4p+Eyx/j4MHFN8P2ocBKgoHFIeHy0A7GVMrBg+ldFgsfHui8sBNxlSQsf43gOjPAdA4e1IwRDGi2+rMFHuLggdMvJxGPEVxLv71Ze0o/rzbiSunnFW5bDAwOl/sDLwMXtdYfcC0HD1zP6WjMHYyrJOEzvR34cYp+98/gwCB/Sj+rg+LqyB+Yvv5FMENkBcG14e9GfKwJ4Q/2HWBJ0/EIrp3+HXgPeC7hl9SAO8PY3gXKE/r6AsEAXiXwLx2M588El0/2ElyTvaorYwHKgcXhPr8ivBm4g3H9MTzuImAuB/8B/W54jOUkzNZp7Wcb/hzeDON9CMhJIqZTCS5/LQLeDr8uTPXn1UZcKf28wv2OBN4KY1gMfK+t/oDc8HVluH5CR2PuYFzPh5/ZYuC/OTDTrNt+98N9z+BAgknpZ5X4pTv5RUQkEhqDERGRSCjBiIhIJJRgREQkEkowIiISCSUYERGJhBKMSCvMrK79rTp9jIs7VKW2c8c8w8xO7s5jSt+U1f4mItIZZpbp7o0trXP3uQT3nHT1MbP8QD2q5s4A6ghu2hWJjM5gRJJgZjeY2fywqOEPE9r/GhYhXZJQiBQzqzOzn5nZO8BHLHimzw/NbGH4zI9p4XafN7Nfhcv3hc8Bec3MYmZ2adieYWZ3mdk/LXh+zBNN65rF+A8zu92CZwZdb2YfD5/78ZaZPWdmw8Pill8EvmbB80s+ambFZvZI+P7mm9kpUX6W0nfoDEakHWZ2HjCZoM6UAXPN7DQPHhHwBXffZGb9gflm9oi71xKU9Znn7t8I+wCocfdjzezLwDeBq1s4XAnBnfbTCM5sHgY+RVAGpwwYRlBe/95Wws129/LwmEOAk9zdzexq4Fvu/g0z+w1Q5+7/FW73AHCbu79iZmOBp4HDOvyBiYSUYETad1749Vb4Oo8g4bwEfNXMPhm2jwnba4FGgmKSiZqKXS4gSBot+asHxSaXmllTGf9TgYfC9vVm9kIbsT6YsDwaeDAsqJlNUPeqJecAZXbgAYr5ZpbnB55/ItIhSjAi7TPgP939/x7UaHYGwR/nj7j7DjP7B0G9J4BdLYy77A6/N9L6v73dCctJPTK3mfqE5V8CP3f3uWGsP2hlnwyCM51dHTieSKs0BiPSvqeBL4TPT8HMRpnZMIJy55vD5DKNoBJuFF4FLgnHYoYTDNIno4AD5dWvTGjfTvCo5CbPAF9pemHhc+VFOksJRqQd7v4MwTPPXzezdwnGRQYBTwFZZrYM+DHwRkQhPEJQIXopQcXehQRPI2zPD4CHzGwBUJPQ/hjwyaZBfuCrQHk4gWEpwSQAkU5TNWWRHqBpTMTMCglKrZ/i7utTHZdIWzQGI9IzPG5mgwkG63+k5CI9gc5gREQkEhqDERGRSCjBiIhIJJRgREQkEkowIiISCSUYERGJhBKMiIhE4v8DbrKdCVzqSvIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 测试\n",
    "temp_learing_rate = CustomSchedule(d_model)\n",
    "plt.plot(temp_learing_rate(tf.range(40000, dtype=tf.float32)))\n",
    "plt.xlabel('learning rate')\n",
    "plt.ylabel('train step')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 损失和指标\n",
    "由于目标序列是填充的，因此在计算损耗时应用填充掩码很重要。\n",
    "padding的掩码为0，没padding的掩码为1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True,\n",
    "                                                           reduction='none')\n",
    "\n",
    "def loss_fun(y_ture, y_pred):\n",
    "    mask = tf.math.logical_not(tf.math.equal(y_ture, 0))  # 为0掩码标1\n",
    "    loss_ = loss_object(y_ture, y_pred)\n",
    "    \n",
    "    mask = tf.cast(mask, dtype=loss_.dtype)\n",
    "    loss_ *= mask\n",
    "    return tf.reduce_mean(loss_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8、训练和保持模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = Transformer(num_layers, d_model, num_heads, dff,\n",
    "                          input_vocab_size, target_vocab_size,\n",
    "                          max_seq_len, dropout_rate)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构建掩码\n",
    "def create_mask(inputs,targets):\n",
    "    encode_padding_mask = create_padding_mark(inputs)\n",
    "    # 这个掩码用于掩输入解码层第二层的编码层输出\n",
    "    decode_padding_mask = create_padding_mark(inputs)\n",
    "    \n",
    "    # look_ahead 掩码， 掩掉未预测的词\n",
    "    look_ahead_mask = create_look_ahead_mark(tf.shape(targets)[1])\n",
    "    # 解码层第一层得到padding掩码\n",
    "    decode_targets_padding_mask = create_padding_mark(targets)\n",
    "    \n",
    "    # 合并解码层第一层掩码\n",
    "    combine_mask = tf.maximum(decode_targets_padding_mask, look_ahead_mask)\n",
    "    \n",
    "    return encode_padding_mask, combine_mask, decode_padding_mask\n",
    "\n",
    "\n",
    "    \n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建checkpoint管理器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_path = './checkpoint/train'\n",
    "ckpt = tf.train.Checkpoint(transformer=transformer,\n",
    "                          optimizer=optimizer)\n",
    "# ckpt管理器\n",
    "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=3)\n",
    "\n",
    "if ckpt_manager.latest_checkpoint:\n",
    "    ckpt.restore(ckpt_manager.latest_checkpoint)\n",
    "    print('last checkpoit restore')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "target分为target_input和target real.\n",
    "target_input是传给解码器的输入，target_real是其左移一个位置的结果，每个target_input位置对应下一个预测的标签\n",
    "\n",
    "如句子=“SOS A丛林中的狮子正在睡觉EOS”\n",
    "\n",
    "target_input =“SOS丛林中的狮子正在睡觉”\n",
    "\n",
    "target_real =“丛林中的狮子正在睡觉EOS”\n",
    "\n",
    "transformer是个自动回归模型：它一次预测一个部分，并使用其到目前为止的输出，决定下一步做什么。\n",
    "\n",
    "在训练期间使用teacher-forcing，即无论模型当前输出什么都强制将正确输出传给下一步。\n",
    "\n",
    "而预测时则根据前一个的输出预测下一个词\n",
    "\n",
    "为防止模型在预期输出处达到峰值，模型使用look-ahead mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def train_step(inputs, targets):\n",
    "    tar_inp = targets[:,:-1]\n",
    "    tar_real = targets[:,1:]\n",
    "    # 构造掩码\n",
    "    encode_padding_mask, combined_mask, decode_padding_mask = create_mask(inputs, tar_inp)\n",
    "    \n",
    "    \n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions, _ = transformer(inputs, tar_inp,\n",
    "                                    True,\n",
    "                                    encode_padding_mask,\n",
    "                                    combined_mask,\n",
    "                                    decode_padding_mask)\n",
    "        loss = loss_fun(tar_real, predictions)\n",
    "    # 求梯度\n",
    "    gradients = tape.gradient(loss, transformer.trainable_variables)\n",
    "    # 反向传播\n",
    "    optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
    "    \n",
    "    # 记录loss和准确率\n",
    "    train_loss(loss)\n",
    "    train_accuracy(tar_real, predictions)\n",
    "    \n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "葡萄牙语用作输入语言，英语是目标语言。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 40, 128)\n",
      "(64, 39, 128)\n",
      "(64, 40, 128)\n",
      "(64, 39, 128)\n",
      "epoch 1, batch 0, loss:4.0259, acc:0.0000\n",
      "epoch 1, batch 500, loss:3.4436, acc:0.0340\n",
      "(31, 40, 128)\n",
      "(31, 39, 128)\n",
      "epoch 1, loss:3.2112, acc:0.0481\n",
      "time in 1 epoch:467.3876633644104 secs\n",
      "\n",
      "epoch 2, batch 0, loss:2.4443, acc:0.0982\n",
      "epoch 2, batch 500, loss:2.3006, acc:0.1139\n",
      "epoch 2, save model at ./checkpoint/train/ckpt-1\n",
      "epoch 2, loss:2.2473, acc:0.1184\n",
      "time in 1 epoch:429.6356120109558 secs\n",
      "\n",
      "epoch 3, batch 0, loss:2.0709, acc:0.1306\n",
      "epoch 3, batch 500, loss:2.0279, acc:0.1412\n",
      "epoch 3, loss:1.9927, acc:0.1443\n",
      "time in 1 epoch:426.3838963508606 secs\n",
      "\n",
      "epoch 4, batch 0, loss:1.8720, acc:0.1571\n",
      "epoch 4, batch 500, loss:1.8020, acc:0.1678\n",
      "epoch 4, save model at ./checkpoint/train/ckpt-2\n",
      "epoch 4, loss:1.7664, acc:0.1714\n",
      "time in 1 epoch:387.37333059310913 secs\n",
      "\n",
      "epoch 5, batch 0, loss:1.6616, acc:0.1807\n",
      "epoch 5, batch 500, loss:1.5908, acc:0.1936\n",
      "epoch 5, loss:1.5610, acc:0.1961\n",
      "time in 1 epoch:389.60524225234985 secs\n",
      "\n",
      "epoch 6, batch 0, loss:1.4435, acc:0.2087\n",
      "epoch 6, batch 500, loss:1.4117, acc:0.2127\n",
      "epoch 6, save model at ./checkpoint/train/ckpt-3\n",
      "epoch 6, loss:1.3852, acc:0.2147\n",
      "time in 1 epoch:438.03212571144104 secs\n",
      "\n",
      "epoch 7, batch 0, loss:1.3070, acc:0.2183\n",
      "epoch 7, batch 500, loss:1.2383, acc:0.2320\n",
      "epoch 7, loss:1.2111, acc:0.2346\n",
      "time in 1 epoch:378.90994358062744 secs\n",
      "\n",
      "epoch 8, batch 0, loss:1.1545, acc:0.2332\n",
      "epoch 8, batch 500, loss:1.0854, acc:0.2508\n",
      "epoch 8, save model at ./checkpoint/train/ckpt-4\n",
      "epoch 8, loss:1.0658, acc:0.2525\n",
      "time in 1 epoch:377.7305886745453 secs\n",
      "\n",
      "epoch 9, batch 0, loss:1.0109, acc:0.2532\n",
      "epoch 9, batch 500, loss:0.9780, acc:0.2645\n",
      "epoch 9, loss:0.9624, acc:0.2656\n",
      "time in 1 epoch:378.3708670139313 secs\n",
      "\n",
      "epoch 10, batch 0, loss:0.9245, acc:0.2576\n",
      "epoch 10, batch 500, loss:0.8940, acc:0.2749\n",
      "epoch 10, save model at ./checkpoint/train/ckpt-5\n",
      "epoch 10, loss:0.8820, acc:0.2757\n",
      "time in 1 epoch:378.5437033176422 secs\n",
      "\n",
      "epoch 11, batch 0, loss:0.8609, acc:0.2728\n",
      "epoch 11, batch 500, loss:0.8305, acc:0.2833\n",
      "epoch 11, loss:0.8222, acc:0.2835\n",
      "time in 1 epoch:378.56798672676086 secs\n",
      "\n",
      "epoch 12, batch 0, loss:0.8031, acc:0.2821\n",
      "epoch 12, batch 500, loss:0.7770, acc:0.2905\n",
      "epoch 12, save model at ./checkpoint/train/ckpt-6\n",
      "epoch 12, loss:0.7700, acc:0.2906\n",
      "time in 1 epoch:378.8077425956726 secs\n",
      "\n",
      "epoch 13, batch 0, loss:0.7457, acc:0.2857\n",
      "epoch 13, batch 500, loss:0.7311, acc:0.2971\n",
      "epoch 13, loss:0.7257, acc:0.2969\n",
      "time in 1 epoch:378.34697437286377 secs\n",
      "\n",
      "epoch 14, batch 0, loss:0.7159, acc:0.2925\n",
      "epoch 14, batch 500, loss:0.6931, acc:0.3026\n",
      "epoch 14, save model at ./checkpoint/train/ckpt-7\n",
      "epoch 14, loss:0.6874, acc:0.3025\n",
      "time in 1 epoch:379.3904767036438 secs\n",
      "\n",
      "epoch 15, batch 0, loss:0.6885, acc:0.2905\n",
      "epoch 15, batch 500, loss:0.6594, acc:0.3072\n",
      "epoch 15, loss:0.6546, acc:0.3070\n",
      "time in 1 epoch:377.10075068473816 secs\n",
      "\n",
      "epoch 16, batch 0, loss:0.6465, acc:0.2961\n",
      "epoch 16, batch 500, loss:0.6306, acc:0.3117\n",
      "epoch 16, save model at ./checkpoint/train/ckpt-8\n",
      "epoch 16, loss:0.6257, acc:0.3115\n",
      "time in 1 epoch:379.0886535644531 secs\n",
      "\n",
      "epoch 17, batch 0, loss:0.6033, acc:0.3021\n",
      "epoch 17, batch 500, loss:0.6023, acc:0.3162\n",
      "epoch 17, loss:0.5984, acc:0.3159\n",
      "time in 1 epoch:377.6911520957947 secs\n",
      "\n",
      "epoch 18, batch 0, loss:0.5469, acc:0.3225\n",
      "epoch 18, batch 500, loss:0.5791, acc:0.3195\n",
      "epoch 18, save model at ./checkpoint/train/ckpt-9\n",
      "epoch 18, loss:0.5755, acc:0.3191\n",
      "time in 1 epoch:378.4746241569519 secs\n",
      "\n",
      "epoch 19, batch 0, loss:0.5287, acc:0.3209\n",
      "epoch 19, batch 500, loss:0.5575, acc:0.3229\n",
      "epoch 19, loss:0.5546, acc:0.3224\n",
      "time in 1 epoch:378.284138917923 secs\n",
      "\n",
      "epoch 20, batch 0, loss:0.5182, acc:0.3193\n",
      "epoch 20, batch 500, loss:0.5374, acc:0.3263\n",
      "epoch 20, save model at ./checkpoint/train/ckpt-10\n",
      "epoch 20, loss:0.5344, acc:0.3257\n",
      "time in 1 epoch:377.9467544555664 secs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 20\n",
    "for epoch in range(EPOCHS):\n",
    "    start = time.time()\n",
    "    \n",
    "    # 重置记录项\n",
    "    train_loss.reset_states()\n",
    "    train_accuracy.reset_states()\n",
    "    \n",
    "    # inputs 葡萄牙语， targets英语\n",
    "    \n",
    "    for batch, (inputs, targets) in enumerate(train_dataset):\n",
    "        # 训练\n",
    "        train_step(inputs, targets)\n",
    "        \n",
    "        if batch % 500 == 0:\n",
    "            print('epoch {}, batch {}, loss:{:.4f}, acc:{:.4f}'.format(\n",
    "            epoch+1, batch, train_loss.result(), train_accuracy.result()\n",
    "            ))\n",
    "            \n",
    "    if (epoch + 1) % 2 == 0:\n",
    "        ckpt_save_path = ckpt_manager.save()\n",
    "        print('epoch {}, save model at {}'.format(\n",
    "        epoch+1, ckpt_save_path\n",
    "        ))\n",
    "    \n",
    "    \n",
    "    print('epoch {}, loss:{:.4f}, acc:{:.4f}'.format(\n",
    "    epoch+1, train_loss.result(), train_accuracy.result()\n",
    "    ))\n",
    "    \n",
    "    print('time in 1 epoch:{} secs\\n'.format(time.time()-start))\n",
    "        \n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
